{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "545f38ff",
   "metadata": {},
   "source": [
    "# 7-3，FM模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f06ffa1c",
   "metadata": {},
   "source": [
    "FM算法全称为因子分解机 (FactorizationMachine)。\n",
    "\n",
    "它是广告和推荐领域非常著名的算法，在线性回归模型上考虑了特征的二阶交互。\n",
    "\n",
    "适合捕捉大规模稀疏特征(类别特征)当中的特征交互。\n",
    "\n",
    "FM及其衍生的一些较有名的算法的简要介绍如下：\n",
    "\n",
    "* FM(FactorizationMachine)：在LR基础上用隐向量点积实现自动化特征二阶交叉，且交互项的计算复杂度是O(n)，效果显著好于LR，速度极快接近LR。\n",
    "\n",
    "* FFM(Field Aware FM): 在FM的基础上考虑对不同的特征域(Field，可以理解成特征的分组)使用不同的隐向量。效果好于FM，但参数量急剧增加，且预测性能急剧下降。\n",
    "\n",
    "* Bilinear-FFM: 双线性FFM。为了减少FFM的参数量，设计共享矩阵来代替针对不同Field的多个隐向量。效果接近FFM，但参数量大大减少，与FM相当。交互后添加LayerNormlization时效果和略好于FFM.\n",
    "\n",
    "* DeepFM: 使用FM模型代替DeepWide中的Wide部分，且FM部分的隐向量与Deep部分的Embedding向量是共享的。FM部分可以捕获二阶显式特征交叉，而Deep部分能够捕获高阶隐式特征组合和交叉。\n",
    "\n",
    "* FiBiNET: 使用SE注意力(Squeeze-and-Excitation)机制来捕获特征重要性，并且使用Bilinear-FFM来捕获二阶特征交互。\n",
    "\n",
    "参考文章：张俊林《FFM及DeepFFM模型在推荐系统的探索》https://zhuanlan.zhihu.com/p/67795161\n",
    "\n",
    "<br>\n",
    "\n",
    "<font color=\"red\">\n",
    " \n",
    "公众号 **算法美食屋** 回复关键词：**pytorch**， 获取本项目源码和所用数据集百度云盘下载链接。\n",
    "    \n",
    "</font> \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4f3e9c86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.__version__=2.4.0\n",
      "torchkeras.__version__=4.0.0\n"
     ]
    }
   ],
   "source": [
    "import torch \n",
    "import torchkeras\n",
    "print(\"torch.__version__=\"+torch.__version__) \n",
    "print(\"torchkeras.__version__=\"+torchkeras.__version__) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78bb862c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2740610c",
   "metadata": {},
   "source": [
    "##  一，FM原理解析\n",
    "\n",
    "FM模型的表达形式如下：\n",
    "\n",
    "$$y_{FM} = x_0 + \\sum_{i=1}^n \\omega_i x_i + \\sum_{i=1}^{n-1}\\sum_{j=i+1}^{n} <\\vec{v_i},\\vec{v_j}> x_i x_j$$\n",
    "\n",
    "其中 前两项与 线性回归一致。\n",
    "\n",
    "第三项为特征交互项。用隐向量的点积来计算交互项的系数。这样做比直接设定一个$n\\times n$的交互参数矩阵$W$的好处是减少了参数数量，参数数量从 $n^2$减少为 $n\\times k$，其中k为隐向量$v_i$的长度。\n",
    "\n",
    "从数学上，FM算法用一组向量的两两内积代替了交互参数矩阵$W$，等价于将对称矩阵W分解成如下形式$W=V^TV$，这也是为什么FM算法被叫做因子分解机。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c34c64fa",
   "metadata": {},
   "source": [
    "非常有意思的是，交互项的计算复杂度也可以由 $O(n^2)$ 降低为 $O(nk)$，这样FM前向推断的计算复杂度近似为线性复杂度。对于特征数量n非常大而稀疏的模型，计算起来毫无压力。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "273f8473",
   "metadata": {},
   "source": [
    "交互项的简化计算类似于 $ab+ac+bc =\\frac{1}{2} ((a+b+c)^2-(a^2+b^2+c^2))$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f47f917",
   "metadata": {},
   "source": [
    "$$\\sum_{i=1}^{n-1}\\sum_{j=i+1}^{n} <\\vec{v_i},\\vec{v_j}> x_i x_j\n",
    "= \\frac{1}{2}(\\sum_{i=1}^{n}\\sum_{j=1}^{n} <\\vec{v_i},\\vec{v_j}> x_i x_j - \\sum_{i=1}^{n} <\\vec{v_i},\\vec{v_i}> x_i x_i)$$\n",
    "$$= \\frac{1}{2}(\\sum_{i=1}^{n}\\sum_{j=1}^{n} \\sum_{f=1}^{k} v_{if}v_{jf} x_i x_j - \\sum_{i=1}^{n} \\sum_{f=1}^{k} v_{if}v_{if} x_i x_i)$$\n",
    "\n",
    "$$= \\frac{1}{2}\\sum_{f=1}^{k}(\\sum_{i=1}^{n}\\sum_{j=1}^{n}  v_{if}v_{jf} x_i x_j - \\sum_{i=1}^{n}  v_{if}v_{if} x_i x_i)$$\n",
    "\n",
    "$$= \\frac{1}{2}\\sum_{f=1}^{k}((\\sum_{i=1}^{n}v_{if}x_i)^2  - \\sum_{i=1}^{n}  (v_{if} x_i)^2)$$\n",
    "\n",
    "可以看到交互项的计算复杂度已经变成 $O(nk)$ 了\n",
    "\n",
    "因此 FM的模型形式也可以改写成：\n",
    "\n",
    "$$y_{FM} = x_0 + \\sum_{i=1}^n \\omega_i x_i +\\frac{1}{2}\\sum_{f=1}^{k}((\\sum_{i=1}^{n}v_{if}x_i)^2  - \\sum_{i=1}^{n}  (v_{if} x_i)^2)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c353cfc",
   "metadata": {},
   "source": [
    "注意到 \n",
    "\n",
    "$$\\frac{\\partial{y_{FM}}}{\\partial{v_{if}}} = (\\sum_{j=1}^{n}v_{jf}x_j) x_i - v_{if}x_i^2$$\n",
    "$$= x_i((\\sum_{j=1}^{n}v_{jf}x_j)  - v_{if}x_i)$$\n",
    "\n",
    "可见，只要训练样本中存在不等于0的 $x_i$ ，就能够给隐向量$\\vec{v_{i}}$贡献梯度，从而学到有效的$\\vec{v_{i}}$表示。\n",
    "\n",
    "同理，只要训练样本中存在不等于0的 $x_j$ ，就能够给隐向量$\\vec{v_{j}}$贡献梯度，从而学到有效的$\\vec{v_{j}}$表示。\n",
    "\n",
    "然后，就可以计算出有意义的交互项的权重$<\\vec{v_{i}},\\vec{v_{j}}>$。\n",
    "\n",
    "这非常重要，这说明非零的交互项权重可以在训练样本中不存在 $x_i$和$x_j$同时不为0的样本的发生。\n",
    "\n",
    "这是FM面对稀疏特征具有很强泛化性的原因。\n",
    "\n",
    "考虑一个典型的给用户推荐商品的推荐场景中，用户所在城市特征和商品类目特征的交互。\n",
    "\n",
    "葫芦岛是一个小城市，渔网是一种小众商品。它们都是稀疏特征，绝大部分样本在这两个onehot位上的取值都是0.\n",
    "\n",
    "稀疏乘以稀疏更加稀疏，所以在训练样本中可能根本不存在葫芦岛城市的用户购买渔网这样的样本。\n",
    "\n",
    "但是只要训练样本中存在着葫芦岛的用户购买其它商品这样的样本，也存在其他城市用户购买渔网这样的样本，FM模型就可以给葫芦岛市的用户购买渔网的可能性作出一个估计，这个值可能不小，最后甚至会给葫芦岛的用户推荐渔网。\n",
    "\n",
    "这就是FM面对稀疏特征具有很强泛化性的一个例子。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a78b3ae6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8a3c6263",
   "metadata": {},
   "source": [
    "## 二，Pytorch代码实现"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d007db0c",
   "metadata": {},
   "source": [
    "下面是FM模型的一个完整pytorch实现。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db12d6bb",
   "metadata": {},
   "source": [
    "$$\\sum_{i=1}^{n-1}\\sum_{j=i+1}^{n} <\\vec{v_i},\\vec{v_j}> x_i x_j = \\sum_{i=1}^{n-1}\\sum_{j=i+1}^{n} <x_i\\vec{v_i},x_j\\vec{v_j}> $$\n",
    "\n",
    "注意的是，我们代码中的embedding向量或者线性层作用结果实际上是 $x_i\\vec{v_i}$ 的结果。这是许多读者包括我在学习FM时候感到困惑的一个地方。\n",
    "\n",
    "对于 离散特征，onehot编码后其 $x_i $ 总是等于1或者0，$x_i$不为0的那些项才会保留到结果中，此时$x_i$总是等于1，因此$x_i\\vec{v_i}$就等于其embedding向量。对于连续特征，通过一个不带偏置的Linear层作用，获取到的实际上就是 $x_i\\vec{v_i}$，包含了$x_i$因子。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9a530e37",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch import nn\n",
    "from torch import nn,Tensor \n",
    "import torch.nn.functional as F \n",
    "\n",
    "class NumEmbedding(nn.Module):\n",
    "    \"\"\"\n",
    "    连续特征用linear层编码\n",
    "    输入shape: [batch_size,features_num(n), d_in], # d_in 通常是1\n",
    "    输出shape: [batch_size,features_num(n), d_out]\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, n: int, d_in: int, d_out: int, bias: bool = False) -> None:\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(Tensor(n, d_in, d_out))\n",
    "        self.bias = nn.Parameter(Tensor(n, d_out)) if bias else None\n",
    "        with torch.no_grad():\n",
    "            for i in range(n):\n",
    "                layer = nn.Linear(d_in, d_out)\n",
    "                self.weight[i] = layer.weight.T\n",
    "                if self.bias is not None:\n",
    "                    self.bias[i] = layer.bias\n",
    "\n",
    "    def forward(self, x_num):\n",
    "        # x_num: batch_size, features_num, d_in\n",
    "        assert x_num.ndim == 3\n",
    "        #x = x_num[..., None] * self.weight[None]\n",
    "        #x = x.sum(-2)\n",
    "        x = torch.einsum(\"bfi,fij->bfj\",x_num,self.weight)\n",
    "        if self.bias is not None:\n",
    "            x = x + self.bias[None]\n",
    "        return x\n",
    "    \n",
    "class CatEmbedding(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征用Embedding层编码\n",
    "    输入shape: [batch_size,features_num], \n",
    "    输出shape: [batch_size,features_num, d_embed]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_embed):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(sum(categories), d_embed)\n",
    "        self.offsets = nn.Parameter(\n",
    "                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)\n",
    "        \n",
    "        torch.nn.init.xavier_uniform_(self.embedding.weight.data)\n",
    "\n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        :param x_cat: Long tensor of size ``(batch_size, features_num)``\n",
    "        \"\"\"\n",
    "        x = x_cat + self.offsets[None]\n",
    "        return self.embedding(x) \n",
    "    \n",
    "class CatLinear(nn.Module):\n",
    "    \"\"\"\n",
    "    离散特征用Embedding实现线性层（等价于先F.onehot再nn.Linear()）\n",
    "    输入shape: [batch_size,features_num], \n",
    "    输出shape: [batch_size,d_out]\n",
    "    \"\"\"\n",
    "    def __init__(self, categories, d_out=1):\n",
    "        super().__init__()\n",
    "        self.fc = nn.Embedding(sum(categories), d_out)\n",
    "        self.bias = nn.Parameter(torch.zeros((d_out,)))\n",
    "        self.offsets = nn.Parameter(\n",
    "                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)\n",
    "\n",
    "    def forward(self, x_cat):\n",
    "        \"\"\"\n",
    "        :param x: Long tensor of size ``(batch_size, features_num)``\n",
    "        \"\"\"\n",
    "        x = x_cat + self.offsets[None]\n",
    "        return torch.sum(self.fc(x), dim=1) + self.bias \n",
    "    \n",
    "    \n",
    "class FMLayer(nn.Module):\n",
    "    \"\"\"\n",
    "    FM交互项\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, reduce_sum=True):\n",
    "        super().__init__()\n",
    "        self.reduce_sum = reduce_sum\n",
    "\n",
    "    def forward(self, x): #注意：这里的x是公式中的 <v_i> * xi\n",
    "        \"\"\"\n",
    "        :param x: Float tensor of size ``(batch_size, num_features, k)``\n",
    "        \"\"\"\n",
    "        square_of_sum = torch.sum(x, dim=1) ** 2\n",
    "        sum_of_square = torch.sum(x ** 2, dim=1)\n",
    "        ix = square_of_sum - sum_of_square\n",
    "        if self.reduce_sum:\n",
    "            ix = torch.sum(ix, dim=1, keepdim=True)\n",
    "        return 0.5 * ix\n",
    "    \n",
    "class FM(nn.Module):\n",
    "    \"\"\"\n",
    "    完整FM模型。\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, d_numerical, categories=None, d_embed=4,\n",
    "                 n_classes = 1):\n",
    "        super().__init__()\n",
    "        if d_numerical is None:\n",
    "            d_numerical = 0\n",
    "        if categories is None:\n",
    "            categories = []\n",
    "        self.categories = categories\n",
    "        self.n_classes = n_classes\n",
    "        \n",
    "        self.num_linear = nn.Linear(d_numerical,n_classes) if d_numerical else None\n",
    "        self.cat_linear = CatLinear(categories,n_classes) if categories else None\n",
    "        \n",
    "        self.num_embedding = NumEmbedding(d_numerical,1,d_embed) if d_numerical else None\n",
    "        self.cat_embedding = CatEmbedding(categories, d_embed) if categories else None\n",
    "        \n",
    "        if n_classes==1:\n",
    "            self.fm = FMLayer(reduce_sum=True)\n",
    "            self.fm_linear = None\n",
    "        else:\n",
    "            assert n_classes>=2\n",
    "            self.fm = FMLayer(reduce_sum=False)\n",
    "            self.fm_linear = nn.Linear(d_embed,n_classes)\n",
    "\n",
    "    def forward(self, x):\n",
    "        \n",
    "        \"\"\"\n",
    "        x_num: numerical features\n",
    "        x_cat: category features\n",
    "        \"\"\"\n",
    "        x_num,x_cat = x\n",
    "        \n",
    "        #linear部分\n",
    "        x = 0.0\n",
    "        if self.num_linear:\n",
    "            x = x + self.num_linear(x_num) \n",
    "        if self.cat_linear:\n",
    "            x = x + self.cat_linear(x_cat)\n",
    "        \n",
    "        #交叉项部分\n",
    "        x_embedding = []\n",
    "        if self.num_embedding:\n",
    "            x_embedding.append(self.num_embedding(x_num[...,None]))\n",
    "        if self.cat_embedding:\n",
    "            x_embedding.append(self.cat_embedding(x_cat))\n",
    "        x_embedding = torch.cat(x_embedding,dim=1)\n",
    "        \n",
    "        if self.n_classes==1:\n",
    "            x = x + self.fm(x_embedding)\n",
    "            x = x.squeeze(-1)\n",
    "        else: \n",
    "            x = x + self.fm_linear(self.fm(x_embedding)) \n",
    "        return x\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3ff538b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 2, 4])\n"
     ]
    }
   ],
   "source": [
    "##测试 NumEmbedding\n",
    "\n",
    "num_embedding = NumEmbedding(2,1,4)\n",
    "x_num = torch.randn(2,2)\n",
    "x_out = (num_embedding(x_num.unsqueeze(-1)))\n",
    "print(x_out.shape)        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ca9ef57f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 3])\n",
      "torch.Size([2, 3, 4])\n"
     ]
    }
   ],
   "source": [
    "##测试 CatEmbedding\n",
    "\n",
    "cat_embedding = CatEmbedding(categories = [3,2,2],d_embed=4) \n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "x_out = cat_embedding(x_cat)\n",
    "print(x_cat.shape)\n",
    "print(x_out.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bb503f2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 3])\n",
      "torch.Size([2, 1])\n"
     ]
    }
   ],
   "source": [
    "##测试 CatLinear\n",
    "\n",
    "cat_linear = CatLinear(categories = [3,2,2],d_out=1) \n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "x_out = cat_linear(x_cat)\n",
    "print(x_cat.shape)\n",
    "print(x_out.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b09022e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 4])\n"
     ]
    }
   ],
   "source": [
    "##测试 FMLayer\n",
    "\n",
    "fm_layer = FMLayer(reduce_sum=False)\n",
    "\n",
    "x = torch.randn(2,3,4)\n",
    "x_out = fm_layer(x)\n",
    "print(x_out.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "050b8e23",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.4033,  1.3612],\n",
       "        [ 2.8410, -4.4903]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##测试 FM\n",
    "\n",
    "fm = FM(d_numerical = 3, categories = [4,3,2],\n",
    "        d_embed = 4,n_classes = 2)\n",
    "self = fm \n",
    "x_num = torch.randn(2,3)\n",
    "x_cat = torch.randint(0,2,(2,3))\n",
    "fm((x_num,x_cat))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29e2b226",
   "metadata": {},
   "source": [
    "## 三，Cretio数据集完整范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b6e89a2",
   "metadata": {},
   "source": [
    "Cretio数据集是一个经典的广告点击率CTR预测数据集。\n",
    "\n",
    "这个数据集的目标是通过用户特征和广告特征来预测某条广告是否会为用户点击。\n",
    "\n",
    "数据集有13维数值特征(I1至I13)和26维类别特征(C14至C39), 共39维特征, 特征中包含着许多缺失值。\n",
    "\n",
    "训练集4000万个样本，测试集600万个样本。数据集大小超过100G.\n",
    "\n",
    "此处使用的是采样100万个样本后的cretio_small数据集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2680b15",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install torchkeras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4ada2b70",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd \n",
    "import datetime \n",
    "\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "import torch \n",
    "from torch import nn \n",
    "from torch.utils.data import Dataset,DataLoader  \n",
    "import torch.nn.functional as F \n",
    "import torchkeras \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21f5b451",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fa3283ff",
   "metadata": {},
   "source": [
    "### 1，准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d72dcaeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder,QuantileTransformer\n",
    "from sklearn.pipeline import Pipeline \n",
    "from sklearn.impute import SimpleImputer \n",
    "\n",
    "dfdata = pd.read_csv(\"./eat_pytorch_datasets/criteo_small.zip\",sep=\"\\t\",header=None)\n",
    "dfdata.columns = [\"label\"] + [\"I\"+str(x) for x in range(1,14)] + [\n",
    "    \"C\"+str(x) for x in range(14,40)]\n",
    "\n",
    "cat_cols = [x for x in dfdata.columns if x.startswith('C')]\n",
    "num_cols = [x for x in dfdata.columns if x.startswith('I')]\n",
    "num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])\n",
    "\n",
    "for col in cat_cols:\n",
    "    dfdata[col]  = LabelEncoder().fit_transform(dfdata[col])\n",
    "\n",
    "dfdata[num_cols] = num_pipe.fit_transform(dfdata[num_cols])\n",
    "\n",
    "categories = [dfdata[col].max()+1 for col in cat_cols]\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ec01fe7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "from torch.utils.data import Dataset,DataLoader \n",
    "\n",
    "#DataFrame转换成torch数据集Dataset, 特征分割成X_num,X_cat方式\n",
    "class DfDataset(Dataset):\n",
    "    def __init__(self,df,\n",
    "                 label_col,\n",
    "                 num_features,\n",
    "                 cat_features,\n",
    "                 categories,\n",
    "                 is_training=True):\n",
    "        \n",
    "        self.X_num = torch.tensor(df[num_features].values).float() if num_features else None\n",
    "        self.X_cat = torch.tensor(df[cat_features].values).long() if cat_features else None\n",
    "        self.Y = torch.tensor(df[label_col].values).float() \n",
    "        self.categories = categories\n",
    "        self.is_training = is_training\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.Y)\n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        if self.is_training:\n",
    "            return ((self.X_num[index],self.X_cat[index]),self.Y[index])\n",
    "        else:\n",
    "            return (self.X_num[index],self.X_cat[index])\n",
    "    \n",
    "    def get_categories(self):\n",
    "        return self.categories\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fb9b516d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dftrain_val,dftest = train_test_split(dfdata,test_size=0.2)\n",
    "dftrain,dfval = train_test_split(dftrain_val,test_size=0.2)\n",
    "\n",
    "ds_train = DfDataset(dftrain,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_val = DfDataset(dfval,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n",
    "ds_test = DfDataset(dftest,label_col = \"label\",num_features = num_cols,cat_features = cat_cols,\n",
    "                    categories = categories, is_training=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cb40472b",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": [
    "dl_train = DataLoader(ds_train,batch_size = 2048,shuffle=True)\n",
    "dl_val = DataLoader(ds_val,batch_size = 2048,shuffle=False)\n",
    "dl_test = DataLoader(ds_test,batch_size = 2048,shuffle=False)\n",
    "\n",
    "for features,labels in dl_train:\n",
    "    break \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3fffe25",
   "metadata": {},
   "source": [
    "### 2，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d6b54af6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------\n",
      "Layer (type)                            Output Shape              Param #\n",
      "==========================================================================\n",
      "Linear-1                                     [-1, 1]                   14\n",
      "Embedding-2                              [-1, 26, 1]            1,296,709\n",
      "NumEmbedding-3                           [-1, 13, 8]                  104\n",
      "Embedding-4                              [-1, 26, 8]           10,373,672\n",
      "FMLayer-5                                    [-1, 1]                    0\n",
      "==========================================================================\n",
      "Total params: 11,670,499\n",
      "Trainable params: 11,670,499\n",
      "Non-trainable params: 0\n",
      "--------------------------------------------------------------------------\n",
      "Input size (MB): 0.000084\n",
      "Forward/backward pass size (MB): 0.002594\n",
      "Params size (MB): 44.519421\n",
      "Estimated Total Size (MB): 44.522099\n",
      "--------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "def create_net():\n",
    "    net = FM(\n",
    "        d_numerical= ds_train.X_num.shape[1],\n",
    "        categories= ds_train.get_categories(),\n",
    "        d_embed = 8, \n",
    "        n_classes = 1\n",
    "    )\n",
    "    return net \n",
    "\n",
    "from torchkeras import summary\n",
    "\n",
    "net = create_net()\n",
    "summary(net,input_data=features);\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "969c88de",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9547e067",
   "metadata": {},
   "source": [
    "### 3，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72f5bc0a-5d9f-44af-904c-275c35b62b98",
   "metadata": {},
   "source": [
    "我们使用梦中情炉torchkeras来实现最优雅的训练循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "17c5c1d4",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel\n",
    "from torchkeras.metrics import AUC\n",
    "\n",
    "net = create_net()\n",
    "loss_fn = nn.BCEWithLogitsLoss()\n",
    "\n",
    "metrics_dict = {\"auc\":AUC()}\n",
    "optimizer = torch.optim.Adam(net.parameters(), lr=0.005, weight_decay=0.001) \n",
    "\n",
    "model = KerasModel(net,\n",
    "                   loss_fn = loss_fn,\n",
    "                   metrics_dict= metrics_dict,\n",
    "                   optimizer = optimizer\n",
    "                  )         \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c621ef79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< 🐌 cpu is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAGJCAYAAABcsOOZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABmiklEQVR4nO3deVhUVR8H8O8wwLDvuyJg4I6aG2mZmpZt5r6k5vpquaRmWWppLiktr4aaSxbWW1luYZtpqampuZuWuygKKgKKgOwwc94/LjMyMuwz3Bn8fp7nPsyce+69vzuI9zfnnnOuQgghQERERFTDrOQOgIiIiB5MTEKIiIhIFkxCiIiISBZMQoiIiEgWTEKIiIhIFkxCiIiISBZMQoiIiEgWTEKIiIhIFkxCiIiISBZMQsjszJkzBwqFArdu3ZI7lBpz5coVKBQKfPnll3KHQiY0fvx4PPnkk3KHUaNGjBgBJycnkx9n27ZtcHJyQkpKismPRcbDJISoyMKFC/HDDz/IHUatlpaWhrFjx8Lb2xuOjo7o0qULjh8/XqFtFQpFqYuhC/ulS5cwePBg+Pj4wN7eHmFhYXj77bdL1Pvkk0/QuHFjqFQq1KlTB1OnTkVWVpZeHW1iXNqyf//+cuOPi4vD559/jpkzZ1bofLOysqBWqytUt7bTaDRYuXIlWrZsCXt7e3h6euKJJ57AyZMndXWefvpphIaGIjIyUsZIqbKs5Q6AyFwsXLgQ/fr1Q69eveQOpVbSaDR47rnncPLkSUybNg1eXl5YsWIFOnfujGPHjiEsLKzM7b/++usSZUePHsWSJUvw1FNP6ZWfOHECnTt3Rp06dfD666/D09MT8fHxSEhI0Kv31ltv4cMPP0S/fv0wefJknDlzBsuWLcPp06fx22+/6er16dMHoaGhJY4/c+ZMZGZmom3btuWe/5IlSxASEoIuXbqUWmf79u1YtWoV/vjjD6SlpUGpVCIkJEQXn5+fX7nHqY1GjRqFtWvXYtiwYZg4cSKysrLw999/Izk5Wa/eyy+/jDfeeANz586Fs7OzTNFSpQgiM/Puu+8KACIlJaVGj+vo6CiGDx9eo8fUiouLEwDEF198Icvxa8L69esFALFx40ZdWXJysnBzcxMvvvhilfY5evRooVAoREJCgq5MrVaLZs2aiYiICJGdnV3qtjdu3BDW1tbipZde0itftmyZACB++umnMo8dHx8vFAqFGDNmTLlx5ufnCy8vL/HOO+8YXJ+ZmSn69u0rFAqFeOaZZ8SyZcvEL7/8IjZs2CBmz54twsLChJubm9i0aVO5xzI3w4cPF46OjlXeXvvvJiYmpty6SUlJQqlUiujo6Cofj2oWkxAyO9ok5OzZs6J///7C2dlZeHh4iEmTJomcnJwS9b/++mvRqlUrYWdnJ9zd3cXAgQNFfHy8Xp0LFy6IPn36CF9fX6FSqUSdOnXEwIEDRVpamhBCCAAlltISkps3bwqlUinmzJlTYt25c+cEALFs2TIhhBC3b98Wr7/+umjWrJlwdHQUzs7O4umnnxYnTpzQ264qSUheXp6YNWuWaNWqlXBxcREODg7iscceE3/88YdevV27dgkAYteuXRU6pvZz9/LyEnZ2dqJBgwZi5syZFY6rNP379xe+vr5CrVbrlY8dO1Y4ODiI3NzcSu0vNzdXuLm5ic6dO+uVb926VQAQv/76qxBCiKysLFFYWFhi+++//14AEFu2bNErT0lJEQDE4MGDyzz+Bx98IACI3bt3lxvrH3/8UWrdgoIC0blzZ1GvXj1x+PBhg9sXFBSIDz74QNja2opffvmlxPqzZ8+Kvn37Cnd3d6FSqUTr1q3Fjz/+qFfniy++EADEnj17xNixY4WHh4dwdnYWL730kkhNTS2xz+XLl4smTZoIW1tb4e/vL8aPHy/u3LlTot7BgwfFM888I9zc3ISDg4MIDw8XUVFRuvXaJOTatWuiZ8+ewtHRUXh5eYnXX3/d4O/lfhEREaJdu3ZCCCnBzMzMLLP+ww8/LF544YVy90vmgX1CyGwNGDAAubm5iIyMxLPPPoulS5di7NixenUWLFiAYcOGISwsDIsXL8aUKVOwc+dOPP7440hLSwMA5Ofno3v37jh48CBeffVVLF++HGPHjsXly5d1db7++muoVCp07NgRX3/9Nb7++mu8/PLLBuPy9fVFp06dsGHDhhLr1q9fD6VSif79+wMALl++jB9++AHPP/88Fi9ejGnTpuHff/9Fp06dcOPGjWp9PhkZGfj888/RuXNnfPDBB5gzZw5SUlLQvXt3nDhxokr7/OeffxAREYE//vgDY8aMwZIlS9CrVy/8/PPPujoFBQW4detWhRaNRqPb7u+//0arVq1gZaX/3067du2QnZ2NCxcuVCrWX3/9FWlpaRgyZIhe+Y4dOwAAKpUKbdq0gaOjIxwcHDBo0CCkpqbq6uXl5QEA7O3t9bZ3cHAAABw7dqzM469duxaBgYF4/PHHy431r7/+gkKhwMMPP1xiXWRkJM6fP4+DBw/qbutoNBpdvxSNRoO0tDS8+eabiIqKwqhRo3D37l3d9qdPn8YjjzyCs2fPYvr06Vi0aBEcHR3Rq1cvbN68ucTxJk6ciLNnz2LOnDkYNmwY1q5di169ekEIoaszZ84cTJgwAQEBAVi0aBH69u2LTz/9FE899RQKCgp09bZv347HH38cZ86cweTJk7Fo0SJ06dIFv/zyi94x1Wo1unfvDk9PT/z3v/9Fp06dsGjRIqxevbrMzy0jIwOHDx9G27ZtMXPmTLi6usLJyQn169c3+PcHAK1bt8Zff/1V5n7JjMidBRHdT9sScv+3mfHjxwsA4uTJk0IIIa5cuSKUSqVYsGCBXr1///1XWFtb68r//vvvErcBDKnM7ZhPP/1UABD//vuvXnmTJk3EE088oXufm5tb4pt/XFycUKlUYt68eXplqGRLSGFhocjLy9Mru3PnjvD19RWjRo3SlVWmJeTxxx8Xzs7O4urVq3p1NRpNif1VZImLi9Nt5+joqBeX1pYtWwQAsW3btgqfuxBC9O3bV6hUqhLfzl944QUBQHh6eoohQ4aITZs2iVmzZglra2vRoUMH3bkcO3ZMABDz58/X237btm0CgHBycir12KdOnRIAxJtvvlmhWIcOHSo8PT1LlKenpwsXFxfxww8/6MpWr14t3N3dBQDRtGlTXYuNVqtWrcTq1at177t27SrCw8P1WpI0Go3o0KGDCAsL05VpW0Jat24t8vPzdeUffvihAKBrOUlOTha2trbiqaee0vu3+8knnwgAYs2aNUII6d9fSEiICAoKKvE7KP7vZfjw4QKA3r93IaQWi9atW5f5uR0/flz3u/T19RUrVqwQa9euFe3atRMKhUJs3bq1xDYLFy4UAERSUlKZ+ybzwJYQMlsTJkzQe//qq68CkL4BA0BMTAw0Gg0GDBig9+3bz88PYWFh2LVrFwDA1dUVAPDbb78hOzvbKLH16dMH1tbWWL9+va7s1KlTOHPmDAYOHKgrU6lUum/+arUat2/fhpOTExo2bFjhUSGlUSqVsLW1BSB9W05NTUVhYSHatGlTpX2npKTgzz//xKhRo1CvXj29dQqFQve6RYsW2L59e4WW4h0pc3JyoFKpShzXzs5Ot76iMjIysGXLFjz77LNwc3PTW5eZmQkAaNu2Lb755hv07dsX8+bNw/z58/HXX39h586dAIBWrVohIiICH3zwAb744gtcuXIFW7duxcsvvwwbG5sy41m7di0AlGiFKc3t27fh7u5eovz333+Hh4cHXnjhBQDA8ePH8fLLL6Nv377YvHkzBg4ciDFjxuht07NnT+zevRsAkJqaij/++AMDBgzA3bt3dX8Dt2/fRvfu3XHx4kVcv35db/uxY8fCxsZG937cuHGwtrbW/V3t2LED+fn5mDJlil6r1ZgxY+Di4oItW7YAkFq24uLiMGXKlBK/g+L/XrReeeUVvfcdO3bE5cuXy/rYdL/L27dv48cff8S4ceMwePBg7Ny5E56ennjvvfdKbKP9nB+kIf6WjKNjyGzdP1rioYcegpWVFa5cuQIAuHjxIoQQpY6q0P5HGxISgqlTp2Lx4sVYu3YtOnbsiBdeeAFDhw7VJSiV5eXlha5du2LDhg2YP38+AOlWjLW1Nfr06aOrp9FosGTJEqxYsQJxcXF6Qy49PT2rdOzi/ve//2HRokU4d+6cXjN5SEhIpfelvSA0a9aszHru7u7o1q1bpfdvb2+vuwVSXG5urm59RX3//ffIzc01mARo9/Piiy/qlQ8ePBgzZszAX3/9pYv/+++/x8CBAzFq1CgAUmI3depU7NmzB+fPnzd4bCEEvv32WzRr1gzNmzevcMyi2O0OrWPHjqFTp066i7b29tpnn30GAOjVqxfUajXmzp2r28bX1xf79u0DAMTGxkIIgVmzZmHWrFkGj5ucnIw6dero3t//9+Lk5AR/f3/d39XVq1cBAA0bNtSrZ2tri/r16+vWX7p0CUD5/14AKdH09vbWK3N3d8edO3fK3E77uwwJCUFERIRezD169MA333yDwsJCWFvfu5RpP2dDiRCZHyYhZDHu/09Fo9FAoVBg69atUCqVJeoXnyBp0aJFGDFiBH788Uf8/vvvmDRpEiIjI3Hw4EHUrVu3SvEMGjQII0eOxIkTJ9CyZUts2LABXbt2hZeXl67OwoULMWvWLIwaNQrz58+Hh4cHrKysMGXKFL3+ElXxzTffYMSIEejVqxemTZsGHx8fKJVKREZG6i4QQOn/GVd1Dor8/Hy9vhVl8fb21v1u/P39kZiYWKKOtiwgIKDCMaxduxaurq54/vnnS6zT7sfX11ev3MfHBwD0Lnx16tTBvn37cPHiRdy8eRNhYWHw8/NDQEAAGjRoYPDY+/fvx9WrVys1H4Wnp6fBC+7t27f1zvvKlSslhvu2a9dO731CQoIugdX+G3rjjTfQvXt3g8c2NLS4phn6+6yI0n6XgPT7LCgoQFZWlt6XCe3nXPzvkMwXkxAyWxcvXtT7Rh8bGwuNRoPg4GAAUsuIEAIhISGlXjCKCw8PR3h4ON555x389ddfePTRR7Fq1Spdk25lvzn16tULL7/8su6WzIULFzBjxgy9Ops2bUKXLl0QHR2tV56Wllbt/yQ3bdqE+vXrIyYmRi/2d999V6+etnla2wlXS/uNVqt+/foApNtKZfnrr7/KnOuiuLi4ON3vq2XLlti7dy80Go1eM/+hQ4fg4OBQod8hICUtu3btwogRIwze3mndujU+++yzErchtB2B7/9GDkitA9oWgjNnziAxMREjRowwePy1a9dCoVBg8ODBFYoXABo1aoS1a9ciPT1d74Lp4uKC9PR03Xs/Pz+9BBKA3i2L3NxcfP3115g9ezaAe78zGxubCrdOXbx4Ue/3l5mZicTERDz77LMAgKCgIADA+fPndfsHpOQzLi5Od5yHHnoIgPTvpSotYxUREBAAPz+/Er9LQPp92tnZlZgPJC4uDl5eXgZ/z2R+2CeEzNby5cv13i9btgwA8MwzzwCQ+mUolUrMnTu3RFO3EAK3b98GIPUfKCws1FsfHh4OKysrvdsDjo6OJS7UZXFzc0P37t2xYcMGrFu3Dra2tiUmOlMqlSVi27hxo8H/VCtL++2y+P4PHTqEAwcO6NULCgqCUqnEn3/+qVe+YsUKvffe3t54/PHHsWbNGsTHx+utK36MqvYJ6devH5KSkhATE6Mru3XrFjZu3IgePXroJRSXLl0qcTHWWrduHTQaTan9MXr27AmVSoUvvvhCr7Xp888/B4Ayp03XaDR488034eDgUKIPAyCNDNq4cSMee+yxEv1mytK+fXsIIUqMuGncuDEOHTqke9+7d29s3rwZy5cvx9WrV/Hrr79i4cKFAIC9e/fiqaeegru7O4YOHQpAag3o3LkzPv30U4OtTIamMF+9erXerbuVK1eisLBQ93fVrVs32NraYunSpXq/9+joaKSnp+O5554DIPWpCQkJQVRUVIm/G0O3nqpq4MCBSEhIwPbt23Vlt27dwo8//ognnniixGirY8eOoX379kY7PpmYHL1hicqiHR0THh4uevToIZYvXy6GDh1qcO6GyMhIAUB06NBBfPjhh2LlypXizTffFGFhYeKjjz4SQgixefNmUadOHTFlyhSxYsUKsXTpUtG2bVthY2MjDhw4oNvXs88+KxwdHcWiRYvEd999Jw4ePFhurN98840AIJydnUWPHj1KrJ89e7YAIEaMGCFWr14tXn31VeHh4SHq168vOnXqpKtXldExa9as0Y0i+vTTT8X06dOFm5ubaNq0qQgKCtKrO2jQIGFtbS2mTp0qli9fLp555hnRunXrEsc8ceKEcHJyEp6enmLGjBli9erVYubMmaJFixYVjqs0hYWF4pFHHhFOTk5i7ty5Yvny5aJp06bC2dlZnDt3Tq9uUFBQiXPQat26tQgICCgx6qi4efPmCQDiySefFMuXLxdjx44VCoWixKRokyZNEmPHjhUrVqwQS5YsEREREUKhUIivvvrK4H5//vlnAUCsWrWqUueel5en+0yLu3btmrC2thbHjx/XlY0bN043usjBwUF89NFHAoCwsrISAwYMKDGJ3+nTp4W7u7vw9PQU06dPF6tXrxbz588Xzz77rGjevLmunnZ0THh4uOjYsaNYtmyZmDhxorCyshKPPfaY3ogW7d/gU089JT755BPx6quvCqVSKdq2bas3smbbtm3CxsZGBAUFiTlz5ohPP/1UvPbaa+Kpp57S1SltsjLtMcpz8+ZN4e/vL5ydncW7774rFi9eLBo0aCDs7e1LzLejnazs888/L3e/ZB6YhJDZ0f7ndObMGdGvXz/h7Ows3N3dxcSJEw1OVvb999+Lxx57TDg6OgpHR0fRqFEjMWHCBHH+/HkhhBCXL18Wo0aNEg899JCws7MTHh4eokuXLmLHjh16+zl37px4/PHHhb29fZmTlRWXkZGhq//NN9+UWJ+bmytef/114e/vL+zt7cWjjz4qDhw4IDp16lTtJESj0YiFCxeKoKAgoVKpxMMPPyx++eUXMXz48BIX8JSUFNG3b1/h4OAg3N3dxcsvv6wbZnr/MU+dOiV69+4t3NzchJ2dnWjYsKGYNWtWheMqS2pqqhg9erTw9PQUDg4OolOnTuLIkSMl6pWWhGgng5s6dWqZx9FoNGLZsmWiQYMGwsbGRgQGBop33nlH7wIqhHRhbtGihW4iua5du5aY7K24QYMGCRsbG3H79u2KnXAxkyZNEqGhoSXKhw8fLiIiIvSGW1+6dEns3btX3LlzR+Tk5IgDBw7oJtYz5NKlS2LYsGHCz89P2NjYiDp16ojnn39eb4bV+ycrc3d3F05OTmLIkCEGz+eTTz4RjRo1EjY2NsLX11eMGzfO4GRl+/btE08++aRwdnYWjo6Oonnz5rrJ+rTnV50kRHt+vXv3Fi4uLsLe3l488cQTBid2W7lypXBwcBAZGRkV2i/JTyGEEdvNiIjIoMuXL6NRo0bYunUrunbtqiu/desWWrdujWbNmuG7776Di4tLiW3VajU2b96Mfv36Vfn4X375JUaOHIkjR46gTZs2Vd6POXv44YfRuXNnfPzxx3KHQhXEPiFERDWgfv36GD16NN5//329ci8vL2zfvh0XLlxAWFgY5s+fj4MHDyI+Ph6nTp3CqlWr0KJFC7zyyisl+urQPdu2bcPFixdLdA4n88aWECIzU5EhsK6urpWaV4PM3927d/HRRx/h888/1+tk6uzsjCFDhmD27Nnw9/ev8v4fhJYQsjwcoktkZioyBPaLL74odQgpWSZnZ2fMmzcPc+fORWxsLG7evAkXFxc0btxYNzMuUW3DlhAiM3Pnzp1yH57WtGnTan0rJiIyB0xCiIiISBbsmEpERESyYJ8QAzQaDW7cuAFnZ2c+BImIiKgShBC4e/cuAgICSsxoez8mIQbcuHEDgYGBcodBRERksRISEsp9QCiTEAO0D0RKSEgwOHEQERERGZaRkYHAwMASDxc0hEmIAdpbMC4uLkxCiIiIqqAi3RnYMZWIiIhkwSSEiCzPiRPAM89IP4nIYjEJISLL8/33wLZtQEyM3JEQUTUwCSEiy/Pzz/o/icgiMQkhIsuSlAScPCm9PnECSE6WNRwiqjomIURkWX77rez3RGQxmIQQkUXRbNkCjVIpvVYqodmyReaIiKiqOE8IEZmX69elWy4G/JGaira//AJntRoAYKVWI+OXX3B0xw484eFheH++vkCdOqaK1uKphcDetDQk5ufD39YWHd3coOTjKqiGMAkhIvMybBjwxx8GVz0BQHPfBdIpOxtPPPlk6fvr2hXYscOIAVaMJVzcY1JSMDk2Ftfy8nRldVUqLAkNRR9vbxkjowcFkxAiMi+vvAIcPw6kpRlcbSVEme/1uLkBL79svNgqyBIu7jEpKeh3+jTu//Su5+Wh3+nT2NS0qdnEClhGUkeVpxCirL/gB1NGRgZcXV2Rnp7OaduJalhaQQGuJiTA49VXEfjrr9AoFGUnGvfR1t/WqRM+nDkThd7ecFYq4WJtDWelslKvVeU8AdSQ0i7u2sulOVzc8zUa1D94ENfz8w2uV0BKmuIeecQsLvSWkNQV96AnTJW5hjIJMYBJCJHpCCGQXFCASzk5iM3JKfHzdmGhrm7/XbuwavFiuGRnw1qjKXffhVZWyHBwwCtTp2Jjly7VjtVGoahU0uJoZYUpsbF653A/HxsbfNu4MdSQkoE8IZCv0SC/gj/zKlFXt81978v/JCWBKhX8bW115+dS7JxLvC9W5lz0087KqkLPDymLJSR1xcWkpGDyxVhcyy+WMNmqsCTM/BImtRrYuxdITAT8/YGOHYGiPt/VwiSkmpiE0IPG2N/cNELgWl5eyUQjNxexOTnILOpYWhpfGxs8ZG+PUHt7hGdl4YU33kDYnj0oKyIBILlrV+RFRyPd0xN31WpkFBbirlpd6dfZFUh4qHzWCoWUlBhIUHTvy0hwHJVKdPz7b9zIy4fBX74AAu3Mq8Wm76nT0pvi4Wik9983M5+EKSYGmDwZuHbtXlndusCSJUCfPtXbd2WuoewTQvSAq2pTd4FGgyu5uboEo3iSEZeTg7wyvt8oIH3LDrW31yUb2p/17ezgbK3/X5Pmsceg3rcP1mUkL2qlEt7t28MqKKjiJ1+KQo0GmUWJSWUSmIs5OTiTnV3u/gNsbeFjawtbhQK2VlYV/qmqbP1S1h1Mu4s+Z0+VG+d/Qx5CA0d7ZBSdX4ZajbtFP7Xnbmjd3aLfU6EQSC0sRGphIVDs31ellZZfKICEvDy0PHIEnjY2sLGygrVCARuFwvBPE65XABj17wUp1vvjtQKgAcb+E4ueT3jJnjDFxAD9+gFCIYAWaYBnPnDbFtdOuaFfPwU2bap+IlJRbAkxgC0h9KAor6l7bePGCHd01CUXxVs14nNzUVZ7hrVCgRA7O/1Eo+h9iL195fpbtGwJcfJkuS0hipYtgb//rvh+jWzHrTt48tTJcuu9mdgCDbLdoVYDnToBDRtK5ZcuAZs2Sc3kajVQWKj/undvoEMHqe65c8CiRSXraH8OHy7VB4CzZ4Fx4+6tT00TuDD3IOCVZ3i2KA2AFBUazn0EHm4KKJVSM721NdC//72+vikpUj9i7TptPSulgLBTo3VHNbr2kBKTlMxCfP6dGgW2hSi0VaPARo0Cm0Lk26iRb10IWzc17D21yYwaSZmFyLYqhNqqdl2iWjo6wldjDydYw1VhAxelNVwU1nC1soar0hoeNjZoUs8a7tbSkpxopcvftLmLQiEtSqXUeqGVkgKDdQHAygrw85N+/8HBwLWQFGBiLOBTLDlMVgHLQxEY5424uKrfmmFLCBGVSy0EJsfGlkhAAOjKBp89W+Y+7K2sDCYZD9nbI1ClgnUVOnaWcPMmcF8Cou18WrzTqgKQpnFPSpLmBqmGggLgzh1pSU2Vltatpf/EAek++qpV99Zp691OcwO+VZV7cf9wqBu0HTPWrLmXhJw7B0yfXnpcQUH3kpCkJODzz0uv2779vdeZmcCePcXXKoBPQoG5p6U4isdadOsAy0Nx/mzJtO/hh/X3a/gZggoA1rAX1pg4SAUASM4FerxRerzDhgH/+5/0OisLcHIC0OIOEFV+Uoc1wUCCA2AtAKUAlJqinwKNwwVG/kegQAgUCoH5kQKFGgFYF9XRbmMt4Bsg0OVJqV6BRoNffxco0AjdvorXt3PWICjk3n4TbhdCOJV9mxEATmRlAcgqu9LNey+t8q2gSbcG7toAd62BzKLlrjUcNDZYOPNewjJrqjVO7rMGMovq5llB+5XCwUH6XPfuLUpA5p4ueVyvPGDOaSS82xR793qjc+fyP/rqYhJCVMupi/XPuFx0y+RSTg5O3L2rdwumNI5WVmjs6KiXYGh/+tvaVrvjYbnum5a9EEpkWDlhWeBovJoQDRd1JqyLt8n89pt0RYP0rfDWrXvJwv1Jw/Dh9xKADRuAN9+Uyu/eLRlGTMy9loUbN4BvvzUUbMUu7s2bKVCvnvRNMzDwXpXAQGDEiJItC9rXLVveqxsSAixYULKO9me7dvfqPvSQdH7aumfOADNnegPvNi35bThF+jaMvd5YsABo0kS/paVRo3tVPT2BFSsMt9yo1UCrVvfq2tsDU6fqry++PProvbpWVkDPnsDVBDecSC4/qXvoYBC8PBQQAiWWx+oB0+rd22TLCSAnp2Q9IaTP7H+T79VtN1r69yAEoNHo123RQv/5if7P3MHNt8pPmHrdrYcTf6iQlFUIjWMBhGMhhFMhNA7ST6VrARx8CpFedEtLY6sBvPOl5T7ZAKbEFisYXbRo5SuALClhyc22wTP/WCM9RwnMuC2tL+W2ESbE4nqil4EKxsfbMQbwdgwZS00N1ctSqxFXLMEonmxcyc1FQTX+zL9t3BgvVrNlobKEAHJzpW9u9iMGwmHrJgiNgBUEvkdvjMMqpMAH3kjGSryCvtgMDRQQUCDtyf7w/H0dAGDlSmD8+NKP8/339+59r18PDBqkv97NDXB3Bzw8gPnzgWeekcovXgR++kkqL76cOyfda0dHA03dSfcu7rt2oUa+ZZZG2yR//XpRv4DwNF2/APzrBoVQoG5dVKtJ3hh27wa6zC761i5gOKl7tyl2zauZb+1l2blboFty+be4dvg+gq6dy/8/QC0E0gsLcaewEGmFhbhTUIA7xd8Xlele31e3/DaZsn2MFpjS2b1K23J0TDUxCSFjMObcBtphrZeL+mTcn2zcLGW+By2bov4ZDxW1YNS3s0O2Wo13rlwp99i7WrRAZ3fD/xkVFkqJQmam9LNePcDOTlp35gxw9Oi9dcXrZWYCs2cDYWFS3f/9T7rIa+tlZUnfPJUoxG14whUZSIUbXsGn2IgBJeLojw1YhZfhgTQUOLjAJiMVUCqxfj0wdKiUIGiTieLLsGH3vq3fuiX1ydCuc3Or/AXYUi7uwL3OiYCU9Glpc+Sa7JxYGl3/hfopwAQDSd2K6vdfMBa1GvDtl4Lbk0pPmDyXNUXSRm+TxyqEQKZarZ+gFCUsO+7cwdoKPHn6m4aNMcS/al8+2CeESGZVmY2yQKPB1dxcXMrN1Us2Lufk4HJubrnDWt2trVH/vkRD+7qOSlWiBSa/UODdY4lQu5fxze2WCkEZbkBRDrJkidQZUpso3H8358gRoE0b6fXPP5fdv2HUqHtJSHa2lADczx45iFPUhzI0BF0vSq0fhmzEAOxGZ2x/6BU0trsi7dDZGf36AQMG3LuwlsXLS1qqQ6mUPqN+/QCFUECcvJe8aWOIipL/gglICcamTYaHaUZFyZ+AAMU/T2+I/V5As7R7Sd0pNyg0CkRtMo/PU6kEVr/kjb5zmpZMmG5JrWCrJ5s+AQEAhUIBZ2trOFtbI/C+dcF2dhVKQurY2ZomuPswCSEysvI6fCoAjD1/Huezs3VDXC/l5iI+N7fMSaS0w1rrF3UAvT/ZcLex0atfUCB9I796CtiXAMTHAwkJ0vLZZ8DZswqol5TTf+GTUOxIV2DMGKk4J0fa/n5KJeDoCBRvkAkLA7p3lzoYOjpKS/HXISH36vbsCTRvXrKOo6MzlDiK3XuVSCln7rEU+ODO5zGw7ajWXZXkuDhZwsVdq08f6bM3xYRVxnLv81TgWrGkLjDQPD/P7+GNSVO8cN0jTZcw1b3jhiUfK8wi1o5ubqirUuFabl6Zc690dHOrkXh4O8YA3o6h6th95w66nKxAj34D7K2sdElF8WSjvr09gu3sdMNaNRppdIQ2qUhIkDpZau+azJ0rLaX9dR88CFy+DAwejHL7L6xcKQ3DBKSLamKifrLg5ATY2lastaGq9G5zGDgnhQJmc5tDy1SzUT6oLOnzNPdYtS21APS+LBlrFlrejiGqIXkaDc5lZ+NkZib+yczEP1lZOJSRUaFtH3VxwRPu7lKLRlGy4WdrC0CBtDQpsQgNkIbWAcB330nDQhMSpGSgoEB/f+3bAxER0mt3d+libWsrXZzr1ZO+OWqXevWkVg0AwF5vYL9Xif4L0Ej/JRUfDVG3rv68BDVF7zaHwnAfBnO5zaGlVMrb+bS2saTP09xj7ePtjU1NmxrssxZVw8/jYRJCVAFCCCTm5+sSjZNFP89lZ6Owio2Jc4NC4HLZHb/+Cuwtas3Q3jLJKppG4MAB4JFHpNcpKcCff97b3spK+palTSrs7e+tGzYMGDgQ8PaW6hni4yMlFNevA0KjAIo1dQP3Whc6dqzS6RmdJd3mIDJ3fby90dPLS/YH7TEJIbpPjlqNM9nZ+glHZmapDyVzs7ZGc0dHNHdyQgtHR+RdcMTElNNSq0IZQ/UANxz4B5gzx3AcXl5A8UaVp5+W5qbQJh3+/sB93UDuxeRW/nlaYuuCJfRhILIUSoWi1JFvNYVJCFkkY8y/IYRAQl6eLtn4JzMTJ7OycCE722AHUSsADR0cdAlHc0dHtHByQl2VSm/Crq92APg8rNwJq5LHKdCmDfCf/9xLLLS3S+rWvXcbRqtBA2kxJktsXTD3pm4iqjgmIWRxqjL/RpZajdPFbqNoE4+0Ulo3PKyt0cLJCS2Kko3mTk5o4uAA+wp85U5IgNTPopzZKP3nSVNwa6fhlgtbF4hILhwdYwBHx5iv8h64tqFJE7R2dta7jfJPVhZic3IMDpm1VijQqKh1o3jCUdHpyFNTpVskKhV0w1hzcwEXl6KOo1bmPWEVEZGxcXQM1UoVeeDagDNnDK4HAF8bG91tlOZFrRyNHBwq9zRXSMPvduyQHjz2ww/S3BiBgdLkW0qlNGPounVFs1Ga+YRVRERyYhJCFmNvWlq5D1wTAJQAmmlbNoolHb621ZsB8NIl4MsvpaV4/4kWLYDRo6UpzLWJhSX2tSAiqmlMQshiJJbzfBStLxo1wkvaZ64b0QcfSDONAtI8HEOGSK0fxR9tXhz7WhARlY1JCFmEhNxcrLpxo0J1A1Wqah1LCODQIeCLL6SRK23bSuWjRwNXr0o/X3jh3oPaysKRHEREpWMSQmYtT6PB4oQEvHf1KrI1ZT1ZReqcWldV9WceJCUBX38t9fU4e/ZeuTYJiYgAfvutSrsmIiIDmISQ2fotNRWvXryIi0Xziz/m6oreXl54o+hxq4aeeRAVGlqp+ULUamDLFinx2LJF6tcBSLOP9u8PvPSSEU6EiIgMYhJCZudqbi5ei43F5lu3AAB+trb4qH59DPH1hUKhQLCdnVGfeTBx4r0nwz7yiNTPY+BAaZgtERGZDpMQMhu5ajX+m5CAhfHxyNFooAQwqW5dzAkOhov1vX+qVX3mQUYGsH498NNPQEyMNOW5Ugm89hpw4wYwciTQpImJT5KIiHQqN0GCiSxfvhzBwcGws7NDREQEDh8+XGrdzp07Q6FQlFiee+45vXpnz57FCy+8AFdXVzg6OqJt27aIj4839alQFW29fRvhR49i1pUryNFo8LirK060aYPFoaF6CYiW9pkHL/r6orO7e6kJiBDAnj3SY+79/ICxY4FffgG2br1X57XXgI8+YgJCRFTTZG8JWb9+PaZOnYpVq1YhIiICUVFR6N69O86fPw8fH58S9WNiYpBfbKjm7du30aJFC/Tv319XdunSJTz22GMYPXo05s6dCxcXF5w+fRp2FRnOQDXqSk4OpsTG4sfbtwEA/ra2+O9DD+FFH58yZyxVq8se+pqcLA2n/eILaX4PrUaNpNst2kfeExGRfGSftj0iIgJt27bFJ598AgDQaDQIDAzEq6++iunTp5e7fVRUFGbPno3ExEQ4OjoCAAYNGgQbGxt8/fXXVYqJ07abXq5ajQ8TEhAZH49cjQbWCgUm16mD2ffdejEkJsbwJGBLltybBOzkSaBlS+m1szMwaJB0u+WRR+7NWkpERMZXmWuorLdj8vPzcezYMXTr1k1XZmVlhW7duuHAgQMV2kd0dDQGDRqkS0A0Gg22bNmCBg0aoHv37vDx8UFERAR++OGHUveRl5eHjIwMvYVM55dbt9D0yBG8e+UKcjUadHFzw8k2bfDfUm69FBcTI02HXjwBAaT3fftK6wFpFtOxY4H//U9qLVm9GmjfngkIEZE5kTUJuXXrFtRqNXx9ffXKfX19cfPmzXK3P3z4ME6dOoX//Oc/urLk5GRkZmbi/fffx9NPP43ff/8dvXv3Rp8+fbBnzx6D+4mMjISrq6tuCQwMrN6JkUGXc3LQ499/0ePUKVzOzUWArS3WNWmCnS1aoElRElkWtVpqASmr7W7SJKkeAHz6KTBsGFCBXRMRkQxk7xNSHdHR0QgPD0e7du10ZZqiCa169uyJ1157DQDQsmVL/PXXX1i1ahU6depUYj8zZszA1KlTde8zMjKYiBhRjlqN9+Pj8UF8PPKEgLVCgal162JWUBCcymn5KG7v3pItIPe7fl2qx1lKiYjMn6xJiJeXF5RKJZKSkvTKk5KS4FfOsz+ysrKwbt06zJs3r8Q+ra2t0eS+oQ6NGzfGvn37DO5LpVJBVc2pvqkkIQR+un0bU2JjcSU3FwDQzd0dy0JD0agKzROJicatR0RE8pL1doytrS1at26NnTt36so0Gg127tyJ9u3bl7ntxo0bkZeXh6FDh5bYZ9u2bXH+/Hm98gsXLiAoKMh4wVOZYrOz8dy//6LXqVO4kpuLuioVNjZpgt+bN69SAgJIo2CMWY+IiOQl++2YqVOnYvjw4WjTpg3atWuHqKgoZGVlYeTIkQCAYcOGoU6dOoiMjNTbLjo6Gr169YKnp2eJfU6bNg0DBw7E448/ji5dumDbtm34+eefsXv37po4pQdatlqNyPh4fBgfj3whYKNQ4PXAQLwTFATHaj4+tmNHaRTM9euG+4UoFNL6jh2rdRgiIqohsichAwcOREpKCmbPno2bN2+iZcuW2LZtm66zanx8PKys9Btszp8/j3379uH33383uM/evXtj1apViIyMxKRJk9CwYUN8//33eOyxx0x+Pg8qIQR+uHULr8XG4mrRdOpPubtjaVgYGjo4GOUYSqU0DLdfPynhKJ6IaEe9REXpzxdCRETmS/Z5QswR5wmpnAvZ2Zh08SJ+u3MHAFBPpcLHoaHo7eVV5oRjlREdDQQHA127Gp4nJDBQSkC084QQEZE8KnMNlb0lhCxXllqNBVevYlFCAvKFgK1CgWmBgZgZFAQHIzZH/Pkn8MorgEYDHDokJRo9e5Y9YyoREZk/JiFUaUIIxBTdekkouvXyjIcHloSGIsxIt160rl0D+vcHCgulWU9bt5bKlUoOwyUisnRMQqhSzmdn49WLF7G96NZLkEqFJWFheMHT02i3XrTy8qRZUJOTgebNgc8/54ynRES1CZMQ0qMWAnvT0pCYnw9/W1t0dHODUqFAZmEh3rt6FYuvXUOBEFApFHirXj28Va+eUW+9FDdxInD4MODuDmzezJlPiYhqGyYhpBOTkoLJsbG4VnSLBQDq2tpigI8PNqSk6Mqf9/REVGgoHrK3N1ksq1dLLR9WVsC6dUD9+iY7FBERyYRJCAGQEpB+p0/j/qFS1/LzsbhoGEqInR2WhIaih5eXyeM5elT6uWAB8NRTJj8cERHJgEkIQS0EJsfGlkhAinNRKvFPmzaVetZLdXz6qTQC5tlna+RwREQkA1mnbSfzsDctTe8WjCEZajWO3r1r0jgKCqRhuIDUAfW559gRlYioNmMSQkjMzzdqvaqaMkVq/UhLM+lhiIjITPB2DMHf1tao9ariiy+AFSuklo/Dh9kPhIjoQcCWEEJHNzfUValKXa8AEKhSoaObm0mOf/QoMG6c9HrOHCYgREQPCiYhBKVCgQkBAQbXabtkRIWGQmmCDhrJydI07Hl5wAsvAO+8Y/RDEBGRmWISQgCAP9PTAQAO9z2xuK5KhU1Nm6KPt7fRj1lQAAwYACQkAA0aAF99Jc0LQkREDwb2CSEcycjA1tRUKAH83aYNbuTllZgx1RRmzgT27AGcnIAffgBcXU1yGCIiMlNMQgjzrl4FAAzx9UUDBwc0MPJD6EozcCCwYQPw8cdA48Y1ckgiIjIjTEIecMfv3sUvt2/DCsDbQUE1euw2bYBz5wATzv5ORERmjHfgH3DzrlwBALzo41MjLSC3bwPHj997zwSEiOjBxSTkAXbi7l38ePs2FADeqYFWkMJCYNAgoEMHYNMmkx+OiIjMHJOQB9j8or4gA3180MjR0eTHe/ttYMcOQKkEGjY0+eGIiMjMMQl5QP2bmYmYW7dqrBVkwwbgww+l1198AYSHm/yQRERk5piEPKC0rSD9vL3R1MStIP/+C4wcKb2eNk2aG4SIiIhJyAPodFYWNqWkAABmmbgV5M4doHdvIDsb6NYNWLjQpIcjIiILwiTkAfTe1asQAPp4eSHcycmkx1qxArh0CQgKAtatA6w5KJyIiIrwkvCAOZeVhfXJyQBM3woCADNmSKNiXngB8PQ0+eGIiMiCMAl5wGhbQXp6eqKls7PJj2dlBbz7rskPQ0REFoi3Yx4gF7Kz8V1RK8js4GCTHefcOWD8eCAnx2SHICKiWoAtIQ+QBVevQgPgeU9PtDJRK0hGBtCrF3D+vNQK8sknJjkMERHVAmwJeUDEZmdjbVISAGC2ifqCaDTAsGFSAlK3LjB7tkkOQ0REtQSTkAfEwvh4qAE84+GBti4uJjnGggXAjz8CKhUQEwP4+JjkMEREVEswCXkAxOXk4KubNwGYrhVky5Z7HVBXrADatjXJYYiIqBZhEvIA0LaCPOXujkdcXY2+/4sXgSFDACGAceOAUaOMfggiIqqFmITUcldzc/FlUSvIuyYaEZOUJE1C1qEDEBVlkkMQEVEtxNExtVzk1asoFAJd3dzQwQStIADw2GPA0aNSXxBbW5McgoiIaiEmIbVYQm4u1piwFSQjA9D2cTXhtCNERFRL8XZMLfZ+fDwKhEBnNzd0dHMz6r5//11KPH75xai7JSKiBwiTkFrqel4ePk9MBAC8a+QRMZcvA4MGSU/I/fFHo+6aiIgeIExCaqkP4uORLwQ6urqikxFbQbKzgT59pASkXTtg2TKj7ZqIiB4wTEJqocS8PKy+cQOA1BdEoVAYZb9CAGPGACdPShORff89YGdnlF0TEdEDiElILfRhQgLyhEAHFxc8YcRWkKgo4NtvpeG4GzdKU7MTERFVFZOQWuZmXh5WmaAV5NAhYNo06fXixcDjjxtlt0RE9ADjEN1a5r8JCcjVaBDh7Iwn3d2Ntt+HHwZefhnIzAQmTjTabomI6AHGJKQWSc7Px0oTtIIA0iRky5cDajVgxN0SEdEDjLdjapFFCQnI1mjQxtkZT3t4VHt/QgDr1gGFhffKlMpq75aIiAgAW0JqjVv5+Vh+/ToAaV6QqraCqNXA3r1AYiJw8CCwdCnwxRfA1q2AFVNWIiIyIiYhtcTia9eQpdGglZMTnvP0rNI+YmKAyZOBa9f0y/39mYAQEZHx8dJSC6QWFGBZUSvI7Cr2BYmJAfr1K5mAAMBXX0nriYiIjIlJSC3w8bVryFSr0cLRES9UoRVErZZaQIQovc6UKVI9IiIiY2ESYuHuFBRgaVHzRVVbQfbuNdwCoiUEkJAg1SMiIjIWJiEWbsm1a8hQq9HM0RG9vLyqtI+i59wZrR4REVFFMAmxYOmFhYjStoIEBcGqiiNi/P2NW4+IiKgizCIJWb58OYKDg2FnZ4eIiAgcPny41LqdO3eGQqEosTz33HMG67/yyitQKBSIiooyUfTyWXrtGtLVajRxcEBfb+8q76djR+k5MKXlMAoFEBgo1SMiIjIW2ZOQ9evXY+rUqXj33Xdx/PhxtGjRAt27d0dycrLB+jExMUhMTNQtp06dglKpRP/+/UvU3bx5Mw4ePIiAgABTn0aNyygsxMdFrSCzqtEKAkgTkC1ZIr2+fzfa91FRnKiMiIiMS/YkZPHixRgzZgxGjhyJJk2aYNWqVXBwcMCaNWsM1vfw8ICfn59u2b59OxwcHEokIdevX8err76KtWvXwsbGpiZOpUZ9cv067hQWopGDA/r7+FR7f336AG+/Ddw/0WrdusCmTdJ6IiIiY5J1srL8/HwcO3YMM2bM0JVZWVmhW7duOHDgQIX2ER0djUGDBsHR0VFXptFo8NJLL2HatGlo2rRpufvIy8tDXl6e7n1GRkYlzqLm3S0sxKKEBADAO0FBUBrpYS5//gncvi09Lffhh6U+IB07sgWEiIhMQ9Yk5NatW1Cr1fD19dUr9/X1xblz58rd/vDhwzh16hSio6P1yj/44ANYW1tj0qRJFYojMjISc+fOrXjgMltx4wZSCwsRZm+PgdXoC1JcYSFw9Kj0esQIoEkTo+yWiIioVLLfjqmO6OhohIeHo127drqyY8eOYcmSJfjyyy8rPGfGjBkzkJ6erlsSiloZzFGWWo3/FmsFsTbSfOqnTgHZ2YCLC9CokVF2SUREVCZZkxAvLy8olUokJSXplSclJcHPz6/MbbOysrBu3TqMHj1ar3zv3r1ITk5GvXr1YG1tDWtra1y9ehWvv/46goODDe5LpVLBxcVFbzFXK69fx62CAjxkZ4fBRugLonXokPSzbVs+J4aIiGqGrJcbW1tbtG7dGjt37tSVaTQa7Ny5E+3bty9z240bNyIvLw9Dhw7VK3/ppZfwzz//4MSJE7olICAA06ZNw2+//WaS86gp2Wo1PipqBXnbiK0gwL0k5JFHjLZLIiKiMsn+FN2pU6di+PDhaNOmDdq1a4eoqChkZWVh5MiRAIBhw4ahTp06iIyM1NsuOjoavXr1gud9z0rx9PQsUWZjYwM/Pz80bNjQtCdjYqtv3EByQQFC7Oww9L5+NNWlTUIiIoy6WyIiolLJnoQMHDgQKSkpmD17Nm7evImWLVti27Ztus6q8fHxsLrvG//58+exb98+/P7773KELIsctRofFLWCzKxXDzZGbAVJTwfOnpVeMwkhIqKaohCirGenPpgyMjLg6uqK9PR0s+kfsuzaNUyKjUU9lQoXIyJga8QkRKMBLl4ETpwABg402m6JiOgBVJlrqOwtIVS+XLUa78fHAwBmBgUZNQEBpI6oDRtKCxERUU3hOAgLsObmTdzIz0ddlQojyhk1REREZCmYhJi5PI0GkUWtIDPq1YPKyK0gQgBjxkjPhsnMNOquiYiIysTbMWbuy5s3cS0vDwG2thhlglaQK1eAzz8HbGyAl182+u6JiIhKxZYQM5av0WDh1asAgOn16sHOBA9x0Q7NbdECsLc3+u6JiIhKxSTEjH118ybi8/LgZ2uL//j7m+QYnB+EiIjkwiTETBVoNFhQ1BfkzcBA2JvoUbZMQoiISC5MQszUN0lJuJKbCx8bG7wcEGCSY+TnA8ePS6+ZhBARUU1jEmKGCjUaLCjqCzItMBAOJmoFOXkSyMsD3N2BsDCTHIKIiKhUTELM0LfJybiUmwsvGxuMq1PHZMeJjQWUSqkVRKEw2WGIiIgM4hBdM1Oo0eC9olaQNwID4WiiVhAAePFFoGdPIDXVZIcgIiIqFZMQM7M+JQUXc3LgaW2NCSbqC1Kcg4O0EBER1TTejjEjaiEw/8oVAMDUwEA4WTNHJCKi2otJiBnZmJyM8zk5cLe2xkQT9gUBgO3bgXbtgIULTXoYIiKiUvGrtpnQCIH5RX1BXqtbFy4mbgXZvx84coRPziUiIvmwJcRMfJ+SgjPZ2XBVKjGpbl2TH4+TlBERkdyYhJgBjRCYV9QKMqVuXbiauBVEiHtJyCOPmPRQREREpWISYgZ+uHULp7Ky4KJUYnINtIJcvAjcuQOoVEDz5iY/HBERkUFMQmSmEQLzikbETKpbF+42NiY/prYVpFUrwNbW5IcjIiIyiEmIzH6+fRsns7LgpFTitRpoBQHYH4SIiMwDkxAZCSEwt6gV5NU6deBRA60gAODiAgQGMgkhIiJ5KYQQQu4gzE1GRgZcXV2Rnp4OFxcXkx3nl1u30OPUKThaWeHKI4/Aq4bvjQjBZ8YQEZFxVeYaypYQmQghMLdoRMyEOnVqPAEBmIAQEZG8mITIZFtqKo7evQsHKyu8HhhYY8fNzJRaQIiIiOTGJEQGxfuCjAsIgE8NtoL85z+Atzfw3Xc1dkgiIiKDmITIYPudOzh09y7srKwwrV69Gj32oUPA7duAj0+NHpaIiKgEJiE1rHgryCsBAfCtwVaQpCTgyhWpL0jbtjV2WCIiIoOYhNSwP9LS8FdGBlQKBd6swb4gwL35QRo3lobpEhERyYlP0a0BaiGwNy0NN/Lz8X58PABgbEAA/FWqGo2Dk5QREZE5YRJiYjEpKZgcG4treXl65eGOjjUeC5MQIiIyJ0xCTCgmJQX9Tp+GoRGxL1+4AE8bG/Tx9q6RWNRq4PBh6TWfnEtEROaAfUJMRC0EJsfGGkxAtKbExkJdQ5N25OYC48cDTz0FNG1aI4ckIiIqE1tCTGRvWlqJWzDFCQAJeXnYm5aGzu7uJo/H0RF4/32TH4aIiKjCqtQSkp6ejtTU1BLlqampyMjIqHZQtUFifr5R6xEREdU2VUpCBg0ahHXr1pUo37BhAwYNGlTtoGoD/wrO/1HRetW1bx9w506NHIqIiKhCqpSEHDp0CF26dClR3rlzZxzSDsF4wHV0c0NdlQqlPSNOASBQpUJHNzeTx5KZCXTqBHh4ADdvmvxwREREFVKlJCQvLw+FhYUlygsKCpCTk1PtoGoDpUKBJaGhAFAiEdG+jwoNhbIGHmV77Big0QB16gB+fiY/HBERUYVUKQlp164dVq9eXaJ81apVaN26dbWDqi36eHtjU9OmqHPfpGR1VSpsatq0xobncn4QIiIyR1UaHfPee++hW7duOHnyJLp27QoA2LlzJ44cOYLff//dqAFauj7e3ujp5YW9aWlIzM+Hv60tOrq51UgLiBaTECIiMkdVSkIeffRRHDhwAB999BE2bNgAe3t7NG/eHNHR0QgLCzN2jBZPqVDUyDDc0hw8KP3kJGVERGROFELU0GxZFiQjIwOurq5IT0+Hi4U/6e3aNSAwEFAqgfR0ab4QIiIiU6nMNbRKLSHxRQ9hK029evWqslsyAe2tmGbNmIAQEZF5qVISEhwcDEUZfRrUanWVAyLjiogAVq8G7OzkjoSIiEhflZKQv//+W+99QUEB/v77byxevBgLFiwwSmBkHHXrAmPGyB0FERFRSVVKQlq0aFGirE2bNggICMBHH32EPn36VDswIiIiqt2M+hTdhg0b4siRI8bcJVXDlSvAihXAP//IHQkREVFJVWoJuf8hdUIIJCYmYs6cORyia0Z+/x2YMAHo2hXYsUPuaIiIiPRVKQlxc3Mr0TFVCIHAwECDD7YjeWjnB+EkZUREZI6qlITs2rVL772VlRW8vb0RGhoKa+sq7ZJMgDOlEhGROavWZGVnzpxBfHw88vPz9cpfeOGFagcmp9owWVl6OuDuDgghPTnX11fuiIiI6EFQmWtolTqmXr58GS1btkSzZs3w3HPPoVevXujVqxd69+6N3r17V3p/y5cvR3BwMOzs7BAREYHDhw+XWrdz585QKBQllueeew6ANFz4rbfeQnh4OBwdHREQEIBhw4bhxo0bVTlVi3XkiJSABAczASEiIvNUpSRk8uTJCA4ORnJyMhwcHHDq1Cn8+eefaNOmDXbv3l2pfa1fvx5Tp07Fu+++i+PHj6NFixbo3r07kpOTDdaPiYlBYmKibjl16hSUSiX69+8PAMjOzsbx48cxa9YsHD9+HDExMTh//rzFt85UFm/FEBGR2RNV4OnpKU6ePCmEEMLFxUWcO3dOCCHEzp07RcuWLSu1r3bt2okJEybo3qvVahEQECAiIyMrtP3HH38snJ2dRWZmZql1Dh8+LACIq1evVmif6enpAoBIT0+vUH1z1KOHEIAQixfLHQkRET1IKnMNrVJLiFqthrOzMwDAy8tLd6sjKCgI58+fr/B+8vPzcezYMXTr1k1XZmVlhW7duuHAgQMV2kd0dDQGDRoExzIejJKeng6FQgE3NzeD6/Py8pCRkaG3WLpvvgF27gT69pU7EiIiIsOqlIQ0a9YMJ0+eBABERETgww8/xP79+zFv3jzUr1+/wvu5desW1Go1fO/rtODr64ubN2+Wu/3hw4dx6tQp/Oc//ym1Tm5uLt566y28+OKLpXaQiYyMhKurq24JDAys8DmYKxcX4IknAD5LkIiIzFWVkpB33nkHGo0GADBv3jzExcWhY8eO+PXXX7F06VKjBliW6OhohIeHo127dgbXFxQUYMCAARBCYOXKlaXuZ8aMGUhPT9ctCQkJpgqZiIiIilRpUo/u3bvrXoeGhuLcuXNITU2Fu7t7mU/XvZ+XlxeUSiWSkpL0ypOSkuDn51fmtllZWVi3bh3mzZtncL02Abl69Sr++OOPMocJqVQqqFSqCsdt7pYvB+LigMGDgVat5I6GiIjIMKM9O8bDw6NSCQgA2NraonXr1ti5c6euTKPRYOfOnWjfvn2Z227cuBF5eXkYOnRoiXXaBOTixYvYsWMHPD09KxWXpVu7Fli0CDh9Wu5IiIiISif79KZTp07F8OHD0aZNG7Rr1w5RUVHIysrCyJEjAQDDhg1DnTp1EBkZqbdddHQ0evXqVSLBKCgoQL9+/XD8+HH88ssvUKvVuv4lHh4esLW1rZkTk0l+PnD8uPT6kUfkjYWIiKgssichAwcOREpKCmbPno2bN2+iZcuW2LZtm66zanx8PKys9Btszp8/j3379uH3338vsb/r16/jp59+AgC0bNlSb92uXbvQuXNnk5yHufjnHyAvD/DwAEJD5Y6GiIiodNWatr22suRp25cvByZOBJ5+Gti6Ve5oiIjoQWPyadvJfHGmVCIishRMQmqZgweln0xCiIjI3DEJqUWys4G7d6XXpUydQkREZDZk75hKxuPgANy4AVy7Bjxgo5KJiMgCsSWkllEogFow6zwRET0AmIQQERGRLJiE1BJCAM2aAb17A8nJckdDRERUPvYJqSViY6Vp2mNjATc3uaMhIiIqH1tCagnt/CCtWgG1fGZ6IiKqJZiE1BKcpIyIiCwNk5BagpOUERGRpWESUgvk5gInT0qv+eRcIiKyFExCaoG//wYKCgAfHyAoSO5oiIiIKoajY2qBggKgQwegTh1psjIiIiJLwCSkFnj8cWD/fmmuECIiIkvB2zG1CFtBiIjIkjAJsXC5ufeenEtERGRJmIRYuO3bpRlS+/aVOxIiIqLKYRJi4Q4dAjQawNVV7kiIiIgqh0mIheMkZUREZKmYhFgwjQY4ckR6zUnKiIjI0jAJsWDnzgEZGYCDA9C0qdzREBERVQ6TEAumfWhdmzaANWd8ISIiC8MkxILxyblERGTJ+P3ZgnXtCmRlAU8+KXckRERElcckxIL17y8tREREloi3Y4iIiEgWbAmxUP/+Kz0rpnFjQKmUOxoiIqLKY0uIhZo7FwgPBz7+WO5IiIiIqoZJiIXSzpTarp28cRAREVUVkxALdP26tCiVQOvWckdDRERUNUxCLJB2fpBmzQBHR3ljISIiqiomIRaIk5QREVFtwCTEAmmTED60joiILBmTEAtTWHjvyblsCSEiIkvGeUIs0KZNwNGjQKNGckdCRERUdUxCLIy1NfDMM9JCRERkyXg7hoiIiGTBJMTCLF4M/PgjkJ0tdyRERETVw9sxFiQjA3jjDUAIICkJcHCQOyIiIqKqY0uIBTlyREpAgoMBHx+5oyEiIqoeJiEWhJOUERFRbcIkxIJwkjIiIqpNmIRYCCHuPTmXLSFERFQbMAmxEFevAsnJgI0N8PDDckdDRERUfUxCLIR2qvYWLQA7O3ljISIiMgYO0bUQ/foB588DaWlyR0JERGQcTEIshEIBNGggdxRERETGw9sxREREJAsmIRbg33+BgQOBzz6TOxIiIiLjMYskZPny5QgODoadnR0iIiJw+PDhUut27twZCoWixPLcc8/p6gghMHv2bPj7+8Pe3h7dunXDxYsXa+JUTOLPP4ENG4CYGLkjISIiMh7Zk5D169dj6tSpePfdd3H8+HG0aNEC3bt3R3JyssH6MTExSExM1C2nTp2CUqlE//79dXU+/PBDLF26FKtWrcKhQ4fg6OiI7t27Izc3t6ZOy6g4SRkREdVGsichixcvxpgxYzBy5Eg0adIEq1atgoODA9asWWOwvoeHB/z8/HTL9u3b4eDgoEtChBCIiorCO++8g549e6J58+b46quvcOPGDfzwww81eGbGw0nKiIioNpI1CcnPz8exY8fQrVs3XZmVlRW6deuGAwcOVGgf0dHRGDRoEBwdHQEAcXFxuHnzpt4+XV1dERERUeo+8/LykJGRobeYi9RUQHsnqV07eWMhIiIyJlmTkFu3bkGtVsPX11ev3NfXFzdv3ix3+8OHD+PUqVP4z3/+oyvTbleZfUZGRsLV1VW3BAYGVvZUTEbbPSYsDPDwkDcWIiIiY5L9dkx1REdHIzw8HO2q2UQwY8YMpKen65aEhAQjRVh9fHIuERHVVrImIV5eXlAqlUhKStIrT0pKgp+fX5nbZmVlYd26dRg9erReuXa7yuxTpVLBxcVFbzEXt28D1tbslEpERLWPrEmIra0tWrdujZ07d+rKNBoNdu7cifbt25e57caNG5GXl4ehQ4fqlYeEhMDPz09vnxkZGTh06FC5+zRHS5cCGRnA8OFyR0JERGRcsk/bPnXqVAwfPhxt2rRBu3btEBUVhaysLIwcORIAMGzYMNSpUweRkZF620VHR6NXr17w9PTUK1coFJgyZQree+89hIWFISQkBLNmzUJAQAB69epVU6dlVPb2ckdARERkfLInIQMHDkRKSgpmz56NmzdvomXLlti2bZuuY2l8fDysrPQbbM6fP499+/bh999/N7jPN998E1lZWRg7dizS0tLw2GOPYdu2bbDj42eJiIjMhkIIIeQOwtxkZGTA1dUV6enpsvYPmTpVmi11xgygb1/ZwiAiIqqwylxDLXp0TG23dy9w7BhQWCh3JERERMbHJMRM5eYCJ09Krzk8l4iIaiMmIWbq77+BggLAxwcICpI7GiIiIuNjEmKmik9SplDIGwsREZEpMAkxU9qH1nGSMiIiqq2YhJgpTtdORES1nezzhFBJ+fn3npjbtq28sRAREZkKkxAzZGsLrF8vdxRERESmxdsxREREJAsmIWboxg2A89gSEVFtxyTEzGg0QOPGgJcXEBcndzRERESmwyTEzJw7B2RkSDOmBgbKHQ0REZHpMAkxM9qhuW3aANbsNkxERLUYkxAzo52kjPODEBFRbcckxMxoW0I4UyoREdV2TELMSFYW8O+/0mu2hBARUW3HJMSMHDsmjY6pU0daiIiIajN2fTQj/v7A9OmAnZ3ckRAREZkekxAzEhYGREbKHQUREVHN4O0YIiIikgVbQsxEaipw+LD09FwPD7mjISIyLY1Gg/z8fLnDoCqwsbGBUqk0yr6YhJiJ3buBvn2B5s2BkyfljoaIyHTy8/MRFxcHjUYjdyhURW5ubvDz84NCoajWfpiEmAntJGWcH4SIajMhBBITE6FUKhEYGAgrK/YKsCRCCGRnZyM5ORkA4O/vX639MQkxE9pJyjg/CBHVZoWFhcjOzkZAQAAcHBzkDoeqwN7eHgCQnJwMHx+fat2aYQpqBgoLgaNHpddMQoioNlOr1QAAW1tbmSOh6tAmkAUFBdXaD5MQM3D6NJCdDbi4AI0byx0NEZHpVbcvAcnLWL8/JiFmQHsrpm1bgLdHiYjoQcFLnhngk3OJiOhBxCTEDLz1FvDZZ0D//nJHQkRkGdRqaWqD776TfhZ1NbEYwcHBiIqKkjsM2XF0jBlo2FBaiIiofDExwOTJwLVr98rq1gWWLAH69DHdcTt37oyWLVsaJXk4cuQIHB0dqx+UhWNLCBERWYyYGKBfP/0EBACuX5fKY2LkiQuQ5tAoLCysUF1vb28OUQaTENn9/DOwfDkQGyt3JERE8snKKn3JzZXqqNVSC4gQJbfXlk2erH9rprR9VtaIESOwZ88eLFmyBAqFAgqFAl9++SUUCgW2bt2K1q1bQ6VSYd++fbh06RJ69uwJX19fODk5oW3bttixY4fe/u6/HaNQKPD555+jd+/ecHBwQFhYGH766acKxaZWqzF69GiEhITA3t4eDRs2xJIlS/TqdO7cGVOmTNEr69WrF0aMGKF7n5eXh7feeguBgYFQqVQIDQ1FdHR0pT6nymISIrPPPgMmTpSSESKiB5WTU+lL375Snb17S7aAFCeEtH7v3ntlwcGG91lZS5YsQfv27TFmzBgkJiYiMTERgYGBAIDp06fj/fffx9mzZ9G8eXNkZmbi2Wefxc6dO/H333/j6aefRo8ePRAfH1/mMebOnYsBAwbgn3/+wbPPPoshQ4YgNTW13Ng0Gg3q1q2LjRs34syZM5g9ezZmzpyJDRs2VOochw0bhu+++w5Lly7F2bNn8emnn8KpKh9WJbBPiIyE4EypREQVlZho3HqV4erqCltbWzg4OMDPzw8AcO7cOQDAvHnz8OSTT+rqenh4oEWLFrr38+fPx+bNm/HTTz9h4sSJpR5jxIgRePHFFwEACxcuxNKlS3H48GE8/fTTZcZmY2ODuXPn6t6HhITgwIED2LBhAwYMGFCh87tw4QI2bNiA7du3o1u3bgCA+vXrV2jb6mASIqOrV4HkZMDaGnj4YbmjISKST2Zm6eu0s4JX9DElxetduVLlkCqsTZs2eu8zMzMxZ84cbNmyBYmJiSgsLEROTk65LSHNmzfXvXZ0dISLi4vuGS3lWb58OdasWYP4+Hjk5OQgPz8fLVu2rPA5nDhxAkqlEp06darwNsbAJERG2laQFi2Aoqn4iYgeSBUZKNKxozQK5vp1w/1CFAppfceOldtvdd0/yuWNN97A9u3b8d///hehoaGwt7dHv379kJ+fX+Z+bGxs9N4rFIoKPWl43bp1eOONN7Bo0SK0b98ezs7O+Oijj3BIe5EBYGVlBXHfh1Z8ynV7mS5C7BMiI05SRkRUcUqlNAwXkBKO4rTvo6LutZwYm62tre7ZN2XZv38/RowYgd69eyM8PBx+fn64YsImmf3796NDhw4YP348Hn74YYSGhuLSpUt6dby9vZFY7D6VWq3GqVOndO/Dw8Oh0WiwZ88ek8VpCJMQGWmT1EcekTcOIiJL0acPsGkTUKeOfnndulK5KecJCQ4OxqFDh3DlyhXcunWr1FaKsLAwxMTE4MSJEzh58iQGDx5coRaNqgoLC8PRo0fx22+/4cKFC5g1axaOHDmiV+eJJ57Ali1bsGXLFpw7dw7jxo1DWlqa3rkNHz4co0aNwg8//IC4uDjs3r270p1bK4tJiEwKC4GTJ6XXbAkhIqq4Pn2kvh67dgHffiv9jIszbQICSLdZlEolmjRpAm9v71L7eCxevBju7u7o0KEDevToge7du6NVq1Ymi+vll19Gnz59MHDgQEREROD27dsYP368Xp1Ro0Zh+PDhGDZsGDp16oT69eujS5cuenVWrlyJfv36Yfz48WjUqBHGjBmDrKqMZ64Ehbj/JhEhIyMDrq6uSE9Ph4uLi8mOc/cucPQo0LlzyaZFIqLaKDc3F3FxcQgJCYGdnZ3c4VAVlfV7rMw1lB1TZeTsDNyXiBIRET0weDuGiIjIjL3yyitwcnIyuLzyyityh1ctbAmRyYsvSmPZ33wTKJr3hoiIqIR58+bhjTfeMLjOlF0GagKTEBmkpgLr1kmv335b3liIiMi8+fj4wMfHR+4wTIK3Y2Rw+LD0MzQU8PSUNxYiIiK5MAmRAZ8XQ0RExCREFtqZUjlJGRERPciYhNQwIe7djmFLCBERPciYhNSw2FipY6pKJT24joiI6EHFJKSGXb8O+PoCDz8M2NrKHQ0RkWVSC4Hdd+7gu6Qk7L5zB2oLmPw7ODgYUVFRcodhVmRPQpYvX47g4GDY2dkhIiICh7X3KkqRlpaGCRMmwN/fHyqVCg0aNMCvv/6qW69WqzFr1iyEhITA3t4eDz30EObPn1/iEcZy6dwZSEwEfvtN7kiIiCxTTEoKgg8eRJeTJzH47Fl0OXkSwQcPIiYlRe7QqJJknSdk/fr1mDp1KlatWoWIiAhERUWhe/fuOH/+vMEx0fn5+XjyySfh4+ODTZs2oU6dOrh69Src3Nx0dT744AOsXLkS//vf/9C0aVMcPXoUI0eOhKurKyZNmlSDZ1c6hQKw8PlliIhkEZOSgn6nT+P+r5XX8/LQ7/RpbGraFH28vWWJjSpP1paQxYsXY8yYMRg5ciSaNGmCVatWwcHBAWvWrDFYf82aNUhNTcUPP/yARx99FMHBwejUqRNaFOtc8ddff6Fnz5547rnnEBwcjH79+uGpp54qt4WlJgghLUREJBFCIEutrtCSUViISRcvlkhAAOjKJsfGIqOwsEL7q0wL+erVqxEQEACNRqNX3rNnT4waNQqXLl1Cz5494evrCycnJ7Rt2xY7duyo8ueyePFihIeHw9HREYGBgRg/fjwyMzN16+fMmYOWLVvqbRMVFYXg4GC9sjVr1qBp06ZQqVTw9/fHxIkTqxyTKcjWEpKfn49jx45hxowZujIrKyt069YNBw4cMLjNTz/9hPbt22PChAn48ccf4e3tjcGDB+Ott96CUqkEAHTo0AGrV6/GhQsX0KBBA5w8eRL79u3D4sWLS40lLy8PeXl5uvcZGRlGOkt9Bw8CvXsDzz0HREeb5BBERBYlW6OB0969RtmXAHAtLw+u+/ZVqH5mx45wLLp2lKd///549dVXsWvXLnTt2hUAkJqaim3btuHXX39FZmYmnn32WSxYsAAqlQpfffUVevTogfPnz6NevXqVPhcrKyssXboUISEhuHz5MsaPH48333wTK1asqPA+Vq5cialTp+L999/HM888g/T0dOzfv7/SsZiSbEnIrVu3oFar4evrq1fu6+uLc+fOGdzm8uXL+OOPPzBkyBD8+uuviI2Nxfjx41FQUIB3330XADB9+nRkZGSgUaNGUCqVUKvVWLBgAYYMGVJqLJGRkZg7d67xTq4Uhw4BSUlAcrLJD0VEREbk7u6OZ555Bt9++60uCdm0aRO8vLzQpUsXWFlZ6bXKz58/H5s3b8ZPP/1UpdaHKVOm6F4HBwfjvffewyuvvFKpJOS9997D66+/jsmTJ+vK2rZtW+lYTMminh2j0Wjg4+OD1atXQ6lUonXr1rh+/To++ugjXRKyYcMGrF27Ft9++y2aNm2KEydOYMqUKQgICMDw4cMN7nfGjBmYOnWq7n1GRgYCAwONHj8nKSMi0udgZYXMjh0rVPfPtDQ8+++/5db7NTwcjxfrK1jWsStjyJAhGDNmDFasWAGVSoW1a9di0KBBsLKyQmZmJubMmYMtW7YgMTERhYWFyMnJQXx8fKWOobVjxw5ERkbi3LlzyMjIQGFhIXJzc5GdnQ0HB4dyt09OTsaNGzd0CZO5ki0J8fLyglKpRFJSkl55UlIS/Ep5rKy/vz9sbGx0t14AoHHjxrh58yby8/Nha2uLadOmYfr06Rg0aBAAIDw8HFevXkVkZGSpSYhKpYJKpTLSmZWkVgN79wI7d0rv27Qx2aGIiCyKQqGo8C2Rpzw8UFelwvW8PIP9QhQA6qpUeMrDA0qFwqhxAkCPHj0ghMCWLVvQtm1b7N27Fx9//DEA4I033sD27dvx3//+F6GhobC3t0e/fv2Qn59f6eNcuXIFzz//PMaNG4cFCxbAw8MD+/btw+jRo5Gfnw8HBwdYWVmV6NNSUFCge21vb1+9k60hsnVMtbW1RevWrbFTe2WG1NKxc+dOtG/f3uA2jz76KGJjY/U6Bl24cAH+/v6wLZp0Izs7G1b3ZbdKpbJEZ6KaEhMDBAcDXboAt25JZaNGSeVERFRxSoUCS0JDAUgJR3Ha91GhoSZJQADAzs4Offr0wdq1a/Hdd9+hYcOGaNWqFQBg//79GDFiBHr37o3w8HD4+fnhypUrVTrOsWPHoNFosGjRIjzyyCNo0KABbty4oVfH29sbN2/e1EtETpw4oXvt7OyM4OBgvWusOZJ1dMzUqVPx2Wef4X//+x/Onj2LcePGISsrCyNHjgQADBs2TK/j6rhx45CamorJkyfjwoUL2LJlCxYuXIgJEybo6vTo0QMLFizAli1bcOXKFWzevBmLFy9G7969a/z8YmKAfv2Aa9f0yxMTpXImIkREldPH2xubmjZFnftar+uqVDUyPHfIkCHYsmUL1qxZo9fXMCwsDDExMThx4gROnjyJwYMHV/nLb2hoKAoKCrBs2TJcvnwZX3/9NVatWqVXp3PnzkhJScGHH36IS5cuYfny5di6datenTlz5mDRokVYunQpLl68iOPHj2PZsmVVislkhMyWLVsm6tWrJ2xtbUW7du3EwYMHdes6deokhg8frlf/r7/+EhEREUKlUon69euLBQsWiMLCQt36jIwMMXnyZFGvXj1hZ2cn6tevL95++22Rl5dX4ZjS09MFAJGenl7l8yosFKJuXe2g3JKLQiFEYKBUj4joQZGTkyPOnDkjcnJyqrWfQo1G7EpNFd/evCl2paaKQo3GSBGWTa1WC39/fwFAXLp0SVceFxcnunTpIuzt7UVgYKD45JNPRKdOncTkyZN1dYKCgsTHH39coeMsXrxY+Pv7C3t7e9G9e3fx1VdfCQDizp07ujorV64UgYGBwtHRUQwbNkwsWLBABAUF6e1n1apVomHDhsLGxkb4+/uLV199tRpnf09Zv8fKXEMVQnDmivtlZGTA1dUV6enpcKnirGK7d0u3YMqza5c0iyoR0YMgNzcXcXFxCAkJgZ2dndzhUBWV9XuszDVU9mnba6vEROPWIyIiqm2YhJiIv79x6xERUe2wdu1aODk5GVyaNm0qd3g1yqLmCbEkHTsCdetKT801dMNLoZDWV3B4PBER1RIvvPACIiIiDK6zsbGp4WjkxSTERJRKYMkSaRSMQqGfiGhHj0VFSfWIiOjB4ezsDGdnZ7nDMAu8HWNCffoAmzYBderol9etK5X36SNPXEREcuOYCMtmrN8fW0JMrE8foGdPacbUxESpD0jHjmwBIaIHk3bG6/z8fIuZ1ZNKys7OBlD920dMQmqAUslhuEREAGBtbQ0HBwekpKTAxsamxAzXZN6EEMjOzkZycjLc3Nz0HqNSFUxCiIioxigUCvj7+yMuLg5Xr16VOxyqIjc3t1Kf81YZTEKIiKhG2draIiwsrEoPdyP53f8g2epgEkJERDXOysqKM6YSR8cQERGRPJiEEBERkSyYhBAREZEs2CfEAO0kLBkZGTJHQkREZFm0186KTGjGJMSAu3fvAgACAwNljoSIiMgy3b17F66urmXWUQjOnVuCRqPBjRs34OzsDIX2QS8PkIyMDAQGBiIhIQEuLi5yh2Px+HkaHz9T4+LnaXwP8mcqhMDdu3cREBBQ7mR0bAkxwMrKCnXr1pU7DNm5uLg8cH88psTP0/j4mRoXP0/je1A/0/JaQLTYMZWIiIhkwSSEiIiIZMEkhEpQqVR49913oVKp5A6lVuDnaXz8TI2Ln6fx8TOtGHZMJSIiIlmwJYSIiIhkwSSEiIiIZMEkhIiIiGTBJISIiIhkwSSEAACRkZFo27YtnJ2d4ePjg169euH8+fNyh1VrvP/++1AoFJgyZYrcoVi069evY+jQofD09IS9vT3Cw8Nx9OhRucOyWGq1GrNmzUJISAjs7e3x0EMPYf78+RV65gcBf/75J3r06IGAgAAoFAr88MMPeuuFEJg9ezb8/f1hb2+Pbt264eLFi/IEa6aYhBAAYM+ePZgwYQIOHjyI7du3o6CgAE899RSysrLkDs3iHTlyBJ9++imaN28udygW7c6dO3j00UdhY2ODrVu34syZM1i0aBHc3d3lDs1iffDBB1i5ciU++eQTnD17Fh988AE+/PBDLFu2TO7QLEJWVhZatGiB5cuXG1z/4YcfYunSpVi1ahUOHToER0dHdO/eHbm5uTUcqfniEF0yKCUlBT4+PtizZw8ef/xxucOxWJmZmWjVqhVWrFiB9957Dy1btkRUVJTcYVmk6dOnY//+/di7d6/codQazz//PHx9fREdHa0r69u3L+zt7fHNN9/IGJnlUSgU2Lx5M3r16gVAagUJCAjA66+/jjfeeAMAkJ6eDl9fX3z55ZcYNGiQjNGaD7aEkEHp6ekAAA8PD5kjsWwTJkzAc889h27duskdisX76aef0KZNG/Tv3x8+Pj54+OGH8dlnn8kdlkXr0KEDdu7ciQsXLgAATp48iX379uGZZ56ROTLLFxcXh5s3b+r97bu6uiIiIgIHDhyQMTLzwgfYUQkajQZTpkzBo48+imbNmskdjsVat24djh8/jiNHjsgdSq1w+fJlrFy5ElOnTsXMmTNx5MgRTJo0Cba2thg+fLjc4Vmk6dOnIyMjA40aNYJSqYRarcaCBQswZMgQuUOzeDdv3gQA+Pr66pX7+vrq1hGTEDJgwoQJOHXqFPbt2yd3KBYrISEBkydPxvbt22FnZyd3OLWCRqNBmzZtsHDhQgDAww8/jFOnTmHVqlVMQqpow4YNWLt2Lb799ls0bdoUJ06cwJQpUxAQEMDPlGoEb8eQnokTJ+KXX37Brl27ULduXbnDsVjHjh1DcnIyWrVqBWtra1hbW2PPnj1YunQprK2toVar5Q7R4vj7+6NJkyZ6ZY0bN0Z8fLxMEVm+adOmYfr06Rg0aBDCw8Px0ksv4bXXXkNkZKTcoVk8Pz8/AEBSUpJeeVJSkm4dMQmhIkIITJw4EZs3b8Yff/yBkJAQuUOyaF27dsW///6LEydO6JY2bdpgyJAhOHHiBJRKpdwhWpxHH320xLDxCxcuICgoSKaILF92djasrPQvA0qlEhqNRqaIao+QkBD4+flh586durKMjAwcOnQI7du3lzEy88LbMQRAugXz7bff4scff4Szs7PunqWrqyvs7e1ljs7yODs7l+hP4+joCE9PT/azqaLXXnsNHTp0wMKFCzFgwAAcPnwYq1evxurVq+UOzWL16NEDCxYsQL169dC0aVP8/fffWLx4MUaNGiV3aBYhMzMTsbGxuvdxcXE4ceIEPDw8UK9ePUyZMgXvvfcewsLCEBISglmzZiEgIEA3goYACCIhBACDyxdffCF3aLVGp06dxOTJk+UOw6L9/PPPolmzZkKlUolGjRqJ1atXyx2SRcvIyBCTJ08W9erVE3Z2dqJ+/fri7bffFnl5eXKHZhF27dpl8P/N4cOHCyGE0Gg0YtasWcLX11eoVCrRtWtXcf78eXmDNjOcJ4SIiIhkwT4hREREJAsmIURERCQLJiFEREQkCyYhREREJAsmIURERCQLJiFEREQkCyYhREREJAsmIURERCQLJiFE9EDYvXs3FAoF0tLS5A6FiIowCSEiIiJZMAkhIiIiWTAJIaIaodFoEBkZiZCQENjb26NFixbYtGkTgHu3SrZs2YLmzZvDzs4OjzzyCE6dOqW3j++//x5NmzaFSqVCcHAwFi1apLc+Ly8Pb731FgIDA6FSqRAaGoro6Gi9OseOHUObNm3g4OCADh064Pz586Y9cSIqFZMQIqoRkZGR+Oqrr7Bq1SqcPn0ar732GoYOHYo9e/bo6kybNg2LFi3CkSNH4O3tjR49eqCgoACAlDwMGDAAgwYNwr///os5c+Zg1qxZ+PLLL3XbDxs2DN999x2WLl2Ks2fP4tNPP4WTk5NeHG+//TYWLVqEo0ePwtramo+tJ5KT3I/xJaLaLzc3Vzg4OIi//vpLr3z06NHixRdf1D0Sfd26dbp1t2/fFvb29mL9+vVCCCEGDx4snnzySb3tp02bJpo0aSKEEOL8+fMCgNi+fbvBGLTH2LFjh65sy5YtAoDIyckxynkSUeWwJYSITC42NhbZ2dl48skn4eTkpFu++uorXLp0SVevffv2utceHh5o2LAhzp49CwA4e/YsHn30Ub39Pvroo7h48SLUajVOnDgBpVKJTp06lRlL8+bNda/9/f0BAMnJydU+RyKqPGu5AyCi2i8zMxMAsGXLFtSpU0dvnUql0ktEqsre3r5C9WxsbHSvFQoFAKm/ChHVPLaEEJHJNWnSBCqVCvHx8QgNDdVbAgMDdfUOHjyoe33nzh1cuHABjRs3BgA0btwY+/fv19vv/v370aBBAyiVSoSHh0Oj0ej1MSEi88aWECIyOWdnZ7zxxht47bXXoNFo8NhjjyE9PR379++Hi4sLgoKCAADz5s2Dp6cnfH198fbbb8PLywu9evUCALz++uto27Yt5s+fj4EDB+LAgQP45JNPsGLFCgBAcHAwhg8fjlGjRmHp0qVo0aIFrl69iuTkZAwYMECuUyeiMjAJIaIaMX/+fHh7eyMyMhKXL1+Gm5sbWrVqhZkzZ+puh7z//vuYPHkyLl68iJYtW+Lnn3+Gra0tAKBVq1bYsGEDZs+ejfnz58Pf3x/z5s3DiBEjdMdYuXIlZs6cifHjx+P27duoV68eZs6cKcfpElEFKIQQQu4giOjBtnv3bnTp0gV37tyBm5ub3OEQUQ1hnxAiIiKSBZMQIiIikgVvxxAREZEs2BJCREREsmASQkRERLJgEkJERESyYBJCREREsmASQkRERLJgEkJERESyYBJCREREsmASQkRERLL4P3GSbxhlqMP/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='11' class='progress-bar-interrupted' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      11.00% [11/100] [05:40<45:54]\n",
       "      <br>\n",
       "      ████████████████████100.00% [79/79] [val_loss=0.4748, val_auc=0.7674]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< val_auc without improvement in 5 epoch,early stopping >>>>>> \n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "dfhistory = model.fit(train_data = dl_train,\n",
    "    val_data = dl_val,\n",
    "    epochs=100,\n",
    "    ckpt_path='checkpoint',\n",
    "    patience=5,\n",
    "    monitor='val_auc',\n",
    "    mode='max',\n",
    "    plot=True,\n",
    "    cpu=True\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "679352f3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "309f81a4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eb0b24c6",
   "metadata": {},
   "source": [
    "### 4，评估模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "24a5e739",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████| 98/98 [00:04<00:00, 24.16it/s, val_auc=0.769, val_loss=0.475]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'val_loss': 0.47518228997989576, 'val_auc': 0.7691175937652588}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(dl_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd74c29e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7fbbd7a3",
   "metadata": {},
   "source": [
    "### 5，使用模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90f929be-f31d-4c24-af2f-b9e75427e1c2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b5720a7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7691173430695828\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "model.eval()\n",
    "dl_test = model.accelerator.prepare(dl_test)\n",
    "with torch.no_grad():\n",
    "    result = torch.cat([model.forward(t[0]) for t in dl_test])\n",
    "\n",
    "preds = F.sigmoid(result)\n",
    "labels = torch.cat([x[-1] for x in dl_test])\n",
    "\n",
    "val_auc = roc_auc_score(labels.numpy(),preds.numpy())\n",
    "print(val_auc)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d58e311",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "71ec3b4b",
   "metadata": {},
   "source": [
    "### 6，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30747dc0",
   "metadata": {},
   "source": [
    "模型最佳权重已经保存在 model.fit(ckpt_path) 传入的参数中了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "55758fb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net_clone = create_net()\n",
    "net_clone.load_state_dict(torch.load(model.ckpt_path))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b62e72ed",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b887ec5a",
   "metadata": {},
   "source": [
    "**如果本书对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果对本书内容理解上有需要进一步和作者交流的地方，欢迎在公众号\"算法美食屋\"下留言。作者时间和精力有限，会酌情予以回复。\n",
    "\n",
    "也可以在公众号后台回复关键字：**加群**，加入读者交流群和大家讨论。\n",
    "\n",
    "![算法美食屋logo.png](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb,md",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
